Re: Submission of transform tests

On Apr 6, 2012, at 2:21 AM, Aryeh Gregor wrote:

> On Fri, Mar 23, 2012 at 5:23 PM, Linss, Peter <peter.linss@hp.com> wrote:
>> Correct. The rules are:
>> Support files in a 'support' directory.
>> Reference files in a 'reference' directory (preferred), or have 'ref-' or 'notref-' filename prefix, or have '-ref' or '-notref' filename suffix (this will be expanded shortly to allow '-ref-' or '-notref-' within the filename).
>> Everything else is considered a test case.
> 
> Okay, thanks.  I guess the "-ref" suffix is allowed before the
> extension, so my current naming convention for refs is fine, right?

Yes.

> Would it be better if I changed it so the files were in a reference/
> directory?

Personally I have a slight preference that they live in a 'reference' directory (symmetric with support files). But the system is fine with either approach for legacy reasons.

> 
> Also, with respect to test numbering: should I just rename
> abspos-1a.html to abspos-001.html, abspos-1b.html to abspos-002.html,
> etc.?  This looks like it matches the convention for the CSS 2.1 test
> suite.  The advantage of the current naming scheme (which I copied
> from Mozilla) is that it allows tests to be grouped.  E.g.,
> background-1*.html all have background-1-ref.html as their ref file,
> and background-2*.html all have background-2-ref.html as their ref
> file.  If they were numbered consecutively, then it would be less
> obvious which tests corresponded to which refs.

If you have a number of tests that are slight variations on the same theme, you can use letter suffixes, but we still want the number padded to 3 digits. 

So abspos-001a.html, abspos-001b.html, … would be fine as well. Then they could each use abspos-001-ref.html (or reference/abspos-001.html or reference/abspos-001-ref.html).

If a large number of tests have a common reference, you could also use a more generic name for the reference and not number it. This would be better if the tests are not closely related and have different numbers. Note that it's also fair to have test-042.html use reference-001.html. There's no need to copy a reference file simply to match its name to its test. But try to be logical in your names.


> 
> If I stick with the number-letter naming convention from Mozilla, I
> can pad to three digits if you prefer, but it's probably never going
> to be needed -- with this convention, two digits would mean ten
> different *ref* files for a single part of the spec.  At least for
> transforms, it's unlikely that this would ever be needed, because it's
> pretty easy to reuse the same ref file for lots of tests.  (In fact,
> green-rect.html is the ref file for a large fraction of my 3D
> transform tests, and could be used for a majority if I wanted to tweak
> them.)  The letter part of the name can be extended indefinitely
> without padding, because after z would come aa, and "foo-1aa.html"
> sorts after "foo-1z.html" naturally because . in ASCII comes before
> all letters.

The three digit padding ensures proper sorting should the number of tests ever exceed 9 (which you never know what's going to happen 10 years from now). Also test names need to be unique across the entire repository, so there could be tests from another suite entirely that use the same base name (like say a test suite for absolute positioning…) so it's impossible to predict how many will use the same name.

> 
> But this is pretty arbitrary and I'm happy to do whichever way makes
> sense to everyone else.
> 
>> Note that we currently expect spec links in test cases to be to the version last published at http://www.w3.org/TR/, not drafts at dev.w3.org/csswg. So you'll have to update those links anyway (but wait until I update Shepherd).
>> 
>> There's been some discussion about allowing links to draft specs and having the build system adjust links on output, but there's nothing in place to support that yet, and the current general feeling is to disallow draft links to encourage more frequent publishing of drafts.
> 
> Okay.
> 
> On Fri, Mar 23, 2012 at 9:35 PM, Peter Linss <peter.linss@hp.com> wrote:
>> Done.
>> 
>> Since the current ED has a significant number of changes to the section
>> anchors and should be updated shortly anyway, I went ahead and used the
>> sections from the current ED. Note however that you still need to have the
>> www.w3.org/TR/ URLs in your tests, but you can use section anchors from the
>> current ED version.
> 
> Thanks!
> 
> On Fri, Mar 23, 2012 at 9:51 PM, Peter Linss <peter.linss@hp.com> wrote:
>> please update your Mercurial config for the test repo as well. Be sure to use the email address <ayg+csswg@aryeh.name> in the [ui] username.
>> 
>> See: http://wiki.csswg.org/tools/hg#setting-up-mercurial-preferences
>> 
>> You can add that setting in the .hg/hgrc file of your local clone of the test repository if you don't want to use that email address for other repositories.
>> 
>> Doing this will allow Shepherd to correlate your Mercurial commits with the same account as used for your Author metadata (right now it's using different accounts for owner and author).
> 
> What e-mail address is it comparing against?  Would it be possible
> instead to change whatever knows about ayg+csswg@aryeh.name to use
> ayg@aryeh.name?  These days I prefer to use just ayg@aryeh.name for
> any human-visible e-mail address, and use plus-addressing only for
> automated mail reception.

In Shepherd, each account has an account name, full name, email address, and url. It does its best to correlate the name used in commit messages with user accounts so searching works. 

At this point, Shepherd has two user accounts for you, the first (username 'AryehGregor') was created by you on the wiki and is using the email address: 'ayg+csswg@aryeh.name'.

In your tests, you used the href of 'http://aryeh.name/' in your author links and the email address 'ayg@aryeh.name' in your commits. The url wasn't known to your account and the email address wasn't used anywhere so Shepherd created a second account (username 'aryeh_gregor') with the email address 'ayg@aryeh.name' (from your Mercurial commit) and the url 'http://aryeh.name/' (from your author links). It was then binding the test owner (the first person to commit a test), the reviser (the person who committed a changeset), and the author to 'aryeh_gregor'.

I manually updated the user ids in Shepherd so that your existing tests have the author, reviser, and owner set to the 'AryehGregor' account (your wiki/mercurial account) and I added the 'http://aryeh.name/' url to that account so it can be correlated to your author links.

I just want to make sure that future commits you make bind to the same account. To do this, you either need to use 'ayg+csswg@aryeh.name' as the email in your Mercurial config, or you need to change your wiki account to use 'ayg@aryeh.name' (which you can do with the 'Update Profile' button in the wiki). 

Either way the email address used in your Mercurial config should match the email address in your wiki account. Alternatively, you can use your wiki account username in your Mercurial config instead of 'full name <email>'. 

> 
> On Fri, Mar 23, 2012 at 10:22 PM, Linss, Peter <peter.linss@hp.com> wrote:
>> Specifically, that file shouldn't have '-ref' in the filename, it's a support file used by a reference, not a reference in itself.
>> 
>> Also, the other .svg files really need to be in a 'support' directory as well. Shepherd does recognize them as support files for the moment, but that's only because it doesn't accept .svg tests yet. Once I turn that on it would think they're supposed to be test files.
>> 
>> I went ahead and moved the support files and updated the URLs.
>> 
>> I also updated all your spec links to w3.org/TR
> 
> Thanks!  Are files in support/ not allowed to have -ref in their name
> either, or will shepherd figure out that they're support files based
> on the directory?  When I first added those support files they were
> only used by one test each, but at this point it might make more sense
> to rename them to something generic anyway.

Currently the code does treat anything in a 'support' directory as a support file. But that could change and that's my code, who knows what code other people may write (although it is in a separate Python library specifically designed to interact with the tests). It's best practice not to mix patterns so please don't use '-ref' in support file names for safety sake. 

> 
>> some of them are still being reported as an error because they don't have an anchors in them. They need to link to a specific section of the spec (or multiple sections if relevant) instead of simply the spec.
> 
> Okay.  No specific section was relevant to those tests, so I just
> linked them to #transform-property:
> 
> http://hg.csswg.org/test/rev/90c09ec77591

That's fine, you can also link to multiple sections if that's more logical for the test (the first link listed in the source is considered 'primary' and should be the most relevant).

> 
> 
> Is there a way to see a list of tests that shepherd has rejected due
> to formatting errors or such?  http://test.csswg.org/shepherd/ seems
> not to list such tests anywhere.

Shepherd's search is quite powerful and lets you search on just about any aspect (if there's something you want to search on but can't, ask me and I'll add it). 

These are the fields you can search on at the moment:
Name (filename minus extension for tests and references, full file name for support files)
Type (test, reference, support)
Specification
Specification Section (from 'help' links)
Status
Author (from the 'author' link)
Owner (first person to commit the file, this will be reassignable in Shepherd soon)
Reviewer (from the reviewer link)
Commenter (person who entered a comment into Shepherd)
Reviser (person who committed a change to the file)
Changeset (assets changed in specific Mercurial changeset)
Title
Assertion
Flag
Whiteboard (generic status comment)
Comment (contents of Mercurial commit message or Shepherd comment)
Content (actual file content, within the first 64k)

Results can be sorted by name, date of last revision, or date of last Shepherd comment.

Any search field with a '+' button after it also allows multiple selection (just click the '+' button).

To see assets you modified that the system filed issues on, try:
http://test.csswg.org/shepherd/search/reviser/AryehGregor/commenter/system/status/issue/

You can also add text in the 'Comment' search field to search for specific content in comments, like:
http://test.csswg.org/shepherd/search/testcase/reviser/AryehGregor/commenter/system/status/issue/comment/Not%20linked/
for tests missing spec links.

The home page does have a link to all the assets you own that are marked as 'Needs Work', if you're logged in.

Peter

Received on Friday, 6 April 2012 18:06:11 UTC