Re: Unifying testsuite policy and getting rid of CSS exceptions

On 09/14/2017 02:23 PM, Geoffrey Sneddon wrote:
>
> 1. Do we put new test suites for CSS WG specs in versioned directories:
> a. Yes
> b. No

On the one hand, if a feature moves across levels then either the test has
to move or its location isn't quite right anymore. (From the CSSWG tooling
perspective, it doesn't matter where the test is, but it's nice when things
match.)

On the other, modules split (and occasionally merge) as well, which creates
the same problem, and that's more likely to affect existing tests if we use
unlevelled directories.

Also, if there is a desire to have per-section sub-directories, that works
better with levelled directories since spec reorganization of a later spec
won't affect test-spec matching in an earlier one.

> 2. Where do new test suites for CSS WG specs go:
> a. At the top level
> b. In css/
> c. Either at the top level or in css/, with no preference between either
>    except for being consistently in one
> d. Up to the editor of the individual spec as to whether they want to
>    use the CSS WG tooling

My preference would be for keeping them all under the css/ subdirectory.

This makes it easier to find all the CSS tests since they're together,
which is useful for things like wpt.fyi.

It also means that the modularization of the CSS specs won't be (literally)
doubling the number of top-level directories in WPT, which probably makes
it easier for anyone who *isn't* looking for CSS tests to navigate the repo.

> 3. Do we move existing tests to match the above:
> a. Yes
> b. No

Yes.

Having inconsistent (time-dependent, author-dependent, etc.) locations
means that tests belonging to a single spec are split across multiple
directories, making it harder to find the tests one is looking for, or
to check for existing tests before adding new ones in order to avoid
duplicate work. Furthermore there is no clearly correct way forward for
someone adding new tests, which makes that process more confusing than
it needs to be imho.

> 4. Where do we run the extra CSS-only lints:
> a. Only in css/
> b. In all directories containing tests for CSS WG specs
> c. Nowhere (and almost certainly break the CSS WG's tooling)

All CSSWG specs, unless someone is volunteering to compile implementation
reports for all of our specs for the next 10 years by some other method.
The CSSWG does not consider "we have 500 tests for module X, that should
be enough" to be a sufficiently comprehensive method for evaluating test
coverage.

~fantasai

Received on Thursday, 21 September 2017 01:56:19 UTC