Re: [csswg-drafts] Test Metadata

The lack of versioning only becomes an issue when we want to move a spec to PR and we have a next level already in development; a quick skim suggests this is maybe 50% of the time?

>From my understanding, the plan was to move steadily over to the tools others are using for implementations reports and move over to following more wpt policies than we are now, and we were doing this by keeping existing versioned directories (from csswg-test) in `css/` with a bunch of extra lints to keep the build system more-or-less working, and putting new directories at the top level without any version suffix.

The lack of versioning is rarely problematic when it comes to building implementations reports even when features have been pushed back at CR to a later level, because in the majority of cases features are sufficiently self-contained that you can exclude all tests for that feature by just excluding one directory; in other cases you need to keep a blacklist of tests. (Of course, maintaining `<link rel=help>` links in tests to point to the newer spec would also be a way of blacklisting stuff, though you run the risk of people writing tests linking to the latest ED regardless of where their feature comes from, but this is a risk with any form of versioning of the tests.)

It's probably worth pointing out the versioning of tests is essentially only useful for getting to REC, and of little value to anyone else (web developers care about interop and what browsers implement, which doesn't always match cleanly to the levels; implementers tend to care about the latest spec, regardless of status). As such, there's only a brief time where it's actually useful to know what tests constitute level n.

I think it bears repeating that there are plenty of other groups who've used test suites in WPT to reach PR (and whence REC), hence basically every issue is something that's happened before.

As for a subset of the questions (I think addressing everything!):

> What do we do (if anything) about the CSS test harness and its ability to find tests for a spec, based on not necessarily sufficiently accurate metadata.

The grouping is essentially a feature of the build system which there's broad agreement to get rid of (indeed, I think the only reason we still have it is for the test harness).

There had been some talk about writing some tool to replace it when it comes to this feature (showing per-section tests and thus also being able to be used for the inline test status markers in specs); I don't think there was such clear consensus that the new tool would actually deal with separate versions of specs (but again, such a feature's usefulness is brief and typically you can do blacklisting quite easily).

That said, I'm unaware of anyone having done any work on any such tool (@gregwhitworth wrote something that grouped based on links, but didn't do what the build system did and plot that relative to the table of contents of the spec), or anyone willing to pay anyone to work on it.

As for the other main feature of the CSS test harness, displaying results, we're getting closer and closer to having daily results on <http://wpt.fyi/>. Note this doesn't handle manual tests (and there's no really sane way to do so, because it's hard to keep them up-to-date with browser changes and test/support file changes, given with any scripting the latter are impossible to detect in general).

> Should we create directories with unversioned shortnames (in addition to the versioned ones we have) and put all new tests there?

No, this seems like the worst possible outcome to me: we should either move everything to unversioned directories (i.e., question 5 above), or do the change for all new tests at a level change.

> Should we move all existing tests to top-level unlevelled directories to be consistent with the rest of wpt? (and lose the build system continuing to reasonably work and hence the test harness, as we'd lose the lints we have to keep the build system working)

This is my preferred outcome. Having spoken to a couple of people, I think this is probably doable despite wpt's normal policy on not moving files around if we do this carefully (i.e., we do it all at once, with advance notice, and a clear mapping between files), given the perceived benefit of doing so.

-- 
GitHub Notification of comment by gsnedders
Please view or discuss this issue at https://github.com/w3c/csswg-drafts/issues/1730#issuecomment-323394930 using your GitHub account

Received on Friday, 18 August 2017 16:07:42 UTC