Re: Unifying testsuite policy and getting rid of CSS exceptions

On Sat, Sep 23, 2017 at 11:31 AM, Philip Jägenstedt <foolip@google.com>
wrote:

> On "unification", here's what I summarized as a compromise in a private
> exchange with Florian:
>
>    - Unversioned directories. Under css/ to make the lint rules simpler,
>    or in the root with a few more lines of lint logic, doesn't matter.
>    - Keep requiring <link rel=help> for CSS WG stuff.
>    - Don't require links to specific spec sections if upstreaming new
>    tests in bulk, because then it just won't happen.
>    - Encourage versioned spec links in documentation, but don't require
>    it, it's easy to fix in bulk on spec level bump.
>
> Having spent a while trying to figure what the status quo is, here's my
understanding:

   - We want to keep http://test.csswg.org/harness/ ("the test harness")
   working for the CSS WG's implementation reports.
   - the test harness relies on the build system to create a test manifest
   (i.e., a list of tests) and to display tests in its "Run tests" UI
   - the build system requires <link rel=help> links to specifications'
   versioned URL including anchors that Shepherd knows about (and any test
   that doesn't do that gets dropped on the floor and vanishes)

The last of these tends to suggest our current policies are woefully
inadequate for keeping http://test.csswg.org/harness/ working, given it'll
only ever show a subset of tests (and a changeable subset, depending on
what anchors current versions of specs include), and we can't lint to
guarantee that it'll show all tests (because of the changeable nature of
it), and even listing that we have a versioned link is hard (what's a
versioned link?).

This would tend to suggest that the final two points of the above
compromise doesn't fulfil the goal of keeping the test harness working.

Requiring versioned URLs in links I expect would add little value, as I'd
expect any implementer writing a test to just be looking at the last ED and
therefore would always link to the latest version of the spec, regardless
of what level a feature was added in (therefore, e.g., if we're trying to
get Level 4 to REC, including a new feature X which doesn't currently have
tests, it seems likely implementers would write tests for feature X with a
Level 5 URL based on the current ED), and therefore someone in preparing
the implementation report would have to go through and review all of the
links to ensure they cover that version.

It also makes any sort of mass-upstreaming very hard unless you add a bogus
likely-to-remain-stable anchor to the link (to, say, #introduction), which
again means anyone preparing the implementation report would have to go
through and review all of the links.

That said, the problems around the anchors could also be resolved by the
CSS WG amending their tooling to put unknown anchors in a new "unknown"
section, but I believe this would involve changes both in the build system
and in the test harness, both of which are as far as I'm aware relatively
unmaintained. Equally, we could treat unversioned URLs as the latest known
version, but that again involves changes in both. (There appears to be
separate code to group tests into spec sections in both the build system
for the index pages and in the test harness for its results display, rather
than the latter relying on the build system for the grouping.)

/Geoffrey