Re: Unifying testsuite policy and getting rid of CSS exceptions

On Sat, Sep 23, 2017 at 11:34 PM Geoffrey Sneddon <me@gsnedders.com> wrote:

> On Sat, Sep 23, 2017 at 11:31 AM, Philip Jägenstedt <foolip@google.com>
> wrote:
>
>> On "unification", here's what I summarized as a compromise in a private
>> exchange with Florian:
>>
>>    - Unversioned directories. Under css/ to make the lint rules simpler,
>>    or in the root with a few more lines of lint logic, doesn't matter.
>>    - Keep requiring <link rel=help> for CSS WG stuff.
>>    - Don't require links to specific spec sections if upstreaming new
>>    tests in bulk, because then it just won't happen.
>>    - Encourage versioned spec links in documentation, but don't require
>>    it, it's easy to fix in bulk on spec level bump.
>>
>> Having spent a while trying to figure what the status quo is, here's my
> understanding:
>
>    - We want to keep http://test.csswg.org/harness/ ("the test harness")
>    working for the CSS WG's implementation reports.
>    - the test harness relies on the build system to create a test
>    manifest (i.e., a list of tests) and to display tests in its "Run tests" UI
>    - the build system requires <link rel=help> links to specifications'
>    versioned URL including anchors that Shepherd knows about (and any test
>    that doesn't do that gets dropped on the floor and vanishes)
>
> The last of these tends to suggest our current policies are woefully
> inadequate for keeping http://test.csswg.org/harness/ working, given
> it'll only ever show a subset of tests (and a changeable subset, depending
> on what anchors current versions of specs include), and we can't lint to
> guarantee that it'll show all tests (because of the changeable nature of
> it), and even listing that we have a versioned link is hard (what's a
> versioned link?).
>

A versioned link is one where some path component begins with "css3" or
some path component ends with a digit. Tweak definition until it's exactly
right.


> This would tend to suggest that the final two points of the above
> compromise doesn't fulfil the goal of keeping the test harness working.
>
> Requiring versioned URLs in links I expect would add little value, as I'd
> expect any implementer writing a test to just be looking at the last ED and
> therefore would always link to the latest version of the spec, regardless
> of what level a feature was added in (therefore, e.g., if we're trying to
> get Level 4 to REC, including a new feature X which doesn't currently have
> tests, it seems likely implementers would write tests for feature X with a
> Level 5 URL based on the current ED), and therefore someone in preparing
> the implementation report would have to go through and review all of the
> links to ensure they cover that version.
>

So far I haven't heard anyone asking test writers to go back and check what
the earliest level they can link to us. This problem is the same as for
versioned directories and it's fine, if someone notices that some tests can
be down-leveled to become part of an implementation report for
not-the-latest level, they can just fix it.


> It also makes any sort of mass-upstreaming very hard unless you add a
> bogus likely-to-remain-stable anchor to the link (to, say, #introduction),
> which again means anyone preparing the implementation report would have to
> go through and review all of the links.
>
> That said, the problems around the anchors could also be resolved by the
> CSS WG amending their tooling to put unknown anchors in a new "unknown"
> section, but I believe this would involve changes both in the build system
> and in the test harness, both of which are as far as I'm aware relatively
> unmaintained. Equally, we could treat unversioned URLs as the latest known
> version, but that again involves changes in both. (There appears to be
> separate code to group tests into spec sections in both the build system
> for the index pages and in the test harness for its results display, rather
> than the latter relying on the build system for the grouping.)
>

Where is http://test.csswg.org/harness/ maintained? It ought to be a 1- or
2-liner fix that we could do at the same time as removing version numbers
from directories.