Unifying testsuite policy and getting rid of CSS exceptions

(BCC'd to www-style, to keep people in the loop, but can we keep all
replies on the one list?)

There was some discussion in the CSS WG a fortnight ago (minutes:
http://www.w3.org/mid/CADhPm3t6F0RGzPsz=sG4JoCyPk6XEX+FJ+kzHyjHjkF_Q_H4tw@mail.gmail.com
under the "Test Metadata" heading) about where new test suites should
go.

When the CSS testsuite was merged into wpt, we did it with the entire
old repository just being put in `css/`, and adding a few extra lints
(all of these to keep the build system working for the sake of keeping
https://test.csswg.org/harness/ working, a requirement of the CSS WG
being okay with the merge; these are a unique filename constraint, a
requirement for <link rel=help> links, support files being in a
support directory, though these don't suffice for all the requirements
the build system).

At that time, we had a few CSS test suites at the top level of WPT:
cssom, cssom-view, and a few Houdini specs. Shortly after the merge,
some people (including, I believe, the editor) decided to merge the
entire cssom and cssom-view test suites into the top level
directories, rather than having them split across multiple places.
Following this, the code that runs https://test.csswg.org/harness/ was
updated to build the CSS testsuite from the root of wpt, instead of
just the css/ directory (though the lints only apply to css/).

Beyond the extra lints, there's one other major difference: the
directories in css/ are versioned (i.e., the directories map to the
versioned short names), which has typically been an anti-goal given
implementors rarely look at anything except the latest ED. However,
given the general policy of not needlessly moving tests, we've mostly
avoided doing that, so we now have some level 4 tests in level 3
folders. Note the CSS build system currently only cares about <link
rel=help> links, so directory names are irrelevant.

The question is where we go from here, and there are three sides to this.

1. What we do with existing tests:

a. We keep them where they are now.

b. We move all CSS WG spec test suites into css/.

c. We move all CSS WG spec test suites to the top-level.


1. Do we remove the versioning from future CSS WG test suites:

a. Yes.

b. No.


2. Where do new test suites go:

a. At the top level.

b. In css/.

c. Up to the editor of the individual spec.


3. Do we move existing tests to match the above?

a. Yes.

b. No.


4. Where do we run the extra CSS lints:

a. Only in css/.

b. In all directories containing tests for CSS WG specs.


As far as I'm aware, the status-quo is 1a, 2a, 3b, 4a. The CSS WG's
preference is 1b, 2b, 3c, and I assume 4b (given without it the CSS
WG's tooling is likely to get pretty broken pretty fast).

There had been some talk about dropping the requirement for <link
rel=help>, but this requires versioned folders (as you need to require
somewhere has the version) for the build system's metadata
requirements, and it would rely on folders within the spec mapping to
anchors within the spec (and anchors aren't stable across versions,
because specs get refactored).

Does anyone have any preferences?

/Geoffrey

Received on Thursday, 14 September 2017 18:24:55 UTC