Re: Unifying testsuite policy and getting rid of CSS exceptions

> On Oct 4, 2017, at 0:25, Geoffrey Sneddon <me@gsnedders.com> wrote:
> 
> Hello CSS WG!
> 
> The conclusion, from the discussion on public-test-infra and others
> around WPT, is that from the WPT project there is a preference for:
> 
> * Unversioned directories, probably under css/ to make the lint rules
> simpler and easier to understand for contributors.
> * Keep requiring <link rel=help> for CSS WG stuff.
> * Require versioned spec links.
> * Don't require links to specific spec sections, but encourage them
> (but don't block on it indefinitely!).
> 
> Peter agreed on IRC to update the build system and test harness so
> that tests in no section and in unknown sections still appear (the
> status quo is that they don't appear anywhere in any of the CSS WG's
> tooling, which is awkward when test anchors change!).
> 
> Is the WG happy with this?

I am.

I also hope the same requirement about rel=help links will be generalized to the rest of wpt, but that's not a blocking condition to do the above.

Also, I believe that while these rules should be applied going forward, we may need to grant a one time exception: when a browser vendor first starts syncing their existing tests with ours, they may have a bunch of legacy tests which do not have rel=help links. I don't think it is practical to require relabeling of these before starting the synchronization (although allowing links without a specific section should make it easier). If they can be labelled, great, but as long as the vendor starts requiring  rel=help links for all new tests going forward, I'd be OK with allowing existing tests to be imported as is.

I think there are three other things we may want to synchronize on (but again, these don't have to block the above):

1) When a test is manual, we currently have multiple ways of declaring it so:
* <meta name="flags" content="interact">
* <meta name="flags" content="animated">
* <meta name="flags" content="userstyle">
* <meta name="flags" content="font">
* -manual suffix after the main filename but before the extension

The different flags imply different reasons for a test being manual, but they all boil down to the test having to be run manually. I do not think the distinction in reason actually serves a useful purpose. We should continue documenting what these means for historical reasons, but I propose that going forward, the CSSWG aligns with the rest of wpt, and adopts the -manual suffix after the filename to identify newly written manual tests (i.e. tests other than reftests or testharness.js tests).

2) Should the CSS tests and the rest of WPT converge on the use of <meta name="flags">

I suggest that we do, and that the only flags that are worth bothering about are "may" and "should". CSSWG test should stop using the rest, and WPT should start using these two.

Not that the rest is entirely meaningless, but given how little benefit we derive from them, they're not worth perpetuating the "the csswg has excessive and annoying metadata requirements that nobody can be bothered to remember" meme for.

3) What about test assertions?

http://web-platform-tests.org/writing-tests/css-metadata.html#test-assertions

They currently are described as optional, and that should stay that way. There should be an expectation that the test reviewer will push back on a test if isn't clear what the test is testing, but <meta name=assert> is only one way to do that. Comments or explicit assertions in tests that use JS are equally fine.

This should not be specific to the CSSWG, and so the test-assertions section should be moved from the CSSWG documentation into the general WPT documentation, identifying <meta name=assert> as one of the options available to make a test self documenting.

****

Hopefully, if we get all that in place, we can stop having distinct requirements for csswg tests and the rest of wpt. As long as we do not lose essential information (and I think the above proposals does not), I think it is worth eliminating the mental overhead of separate requirements.

—Florian

Received on Wednesday, 4 October 2017 01:23:11 UTC