Re: Towards a better testsuite: Metadata

On 04/13/2016 07:50 AM, ishida@w3.org wrote:
> On 12/04/2016 20:10, fantasai wrote:
>> <link rel="help" href="http://www.w3.org/TR/...">
>> <link rel="help" href="http://www.w3.org/TR/...">
>
> actually, having to add this specific help metadata is always one of the things that most slows me down and irks me when
> creating tests. It's certainly important information, but can't we make the link automatically by using a directory structure
> that matches the document structure.  That's something that would make a real difference to the ease of writing tests.
>
> ie. all tests for the section "7.1. Text Alignment: the text-align shorthand" get added to a subdirectory at
>
> css-text-3/justification/text-align-property/
>
> the tools can then use the directory structure to identify the section to which the test belongs, and bingo you've reduced the
> work of writing tests to a far greater degree than worrying about one meta element.
>
> it also has advantages if the document structure changes. Just move the tests to a different directory - no need to change the
> metadata or update the file contents everywhere.
>
> it also makes it easier for people to locate tests for a particular feature by looking in the repo, rather than having to find
> other locations that provide such info (if there are any) and tracking down tests from there.
>
> it also removes the question of whether my test should point to the TR version of the document or the ED, and btw ask yourself
> what's the current ED location today.
>
> maybe i'm missing something, but handling this automatically seems like a real effort-saver for people creating tests.

Well, there are two problems with this
   * We have tests that test the interaction among sections.
     These are quite important; they also don't fit into a
     hierarchical structure.
   * This requires everyone who uses our tests to mirror
     our exact file structure.
   * It centralizes control over the file structure to our
     mapping file, making that a blocker for adding new
     directories in the test suite.

Wrt document structure changes -- the CSSWG is in the habit
of providing explicit ID markers to all headings in our spec
sources, and keeping those stable even as the spec structure
changes. So that's not really an issue here.

Now, we have had some concerns about too many test files in
one directory, and making it easier to find tests. So I'm
perfectly happy if we want to set up subdirectories to
organize the tests according to their "primary" section.

I'm also alright if we want to associate a default link to
that section with, e.g. a LINK file containing the URL in
that directory. The build system can auto-inject the help
link when building the test suite, and everyone else can
look at the LINK file for the directory. (That said, I'm
a little concerned however that the appropriate granularity
for directory structure will be coarser than subsections.)

Not really OK with creating a canonical directory structure
and then relying on that as the sole information for
associating tests and spec sections, for the reasons above.

Wrt TR vs ED -- Links should be to the TR version, and we can
fix that up once in awhile if people forget. (Also, going
forward, Echidna will allow us to keep TR much more up-to-date,
so there will (hopefully) be no reason to continue to use EDs
for day-to-date use.)

~fantasai

Received on Wednesday, 13 April 2016 17:24:00 UTC