Re: Unifying testsuite policy and getting rid of CSS exceptions

On Mon, Sep 25, 2017 at 6:31 PM, Philip Jägenstedt <foolip@google.com>
wrote:

> On Sun, Sep 24, 2017 at 3:48 AM Geoffrey Sneddon <me@gsnedders.com> wrote:
>
>> On Sun, Sep 24, 2017 at 12:23 AM, Philip Jägenstedt <foolip@google.com>
>> wrote:
>>
>>> On Sat, Sep 23, 2017 at 11:34 PM Geoffrey Sneddon <me@gsnedders.com>
>>> wrote:
>>>
>>>> That said, the problems around the anchors could also be resolved by
>>>> the CSS WG amending their tooling to put unknown anchors in a new "unknown"
>>>> section, but I believe this would involve changes both in the build system
>>>> and in the test harness, both of which are as far as I'm aware relatively
>>>> unmaintained. Equally, we could treat unversioned URLs as the latest known
>>>> version, but that again involves changes in both. (There appears to be
>>>> separate code to group tests into spec sections in both the build system
>>>> for the index pages and in the test harness for its results display, rather
>>>> than the latter relying on the build system for the grouping.)
>>>>
>>>
>>> Where is http://test.csswg.org/harness/ maintained? It ought to be a 1-
>>> or 2-liner fix that we could do at the same time as removing version
>>> numbers from directories.
>>>
>>
>> It's all in http://hg.csswg.org/dev/harness/. It's also far from a
>> 1-or-2-liner fix.
>>
>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/
>> pages/ResultsPage.php#l433 is the code that generates the results table,
>> and the codepaths using writeSectionRows are the harder ones to fix (and
>> the default codepath).
>>
>> You also need to change https://github.com/w3c/
>> csswg-w3ctestlib/blob/master/Indexer.py to put unknown anchors somewhere
>> in the index in the first place and then change http://hg.csswg.org/
>> dev/harness/file/3e982ede4ca5/python/SynchronizeSpecLinks.py so that
>> these reach the database the test harness works from.
>>
>> And I've probably missed something, because that's all I found looking
>> last night without looking overly hard.
>>
>
> Alan, Florian, other CSS WG folks, is there anyone that understands this
> code still? It seems to me that even if we try very hard to not break
> what's currently working, then over time it would break as sections change
> names, or as typos occur. Without a tool that can automatically check the
> whole test suite for being in the correct shape, it will be hard to keep it
> from bitrotting.
>
> On the other hand, if such a tool did exist, then I think there's a pretty
> easy fix here: make a best effort to keep things in good shape with the
> line, but run the tool itself when it's needed to take a spec to REC and
> fix the problems it reports then.
>

Having spoken to him on IRC, as I understand it Peter (Linss) is willing to
amend it and the build system to create an "unknown" section in both the
build output and the harness (and he's the one who's touched both most
since the CSS WG adopted the system from whichever group had originally
created it).

Adding a "dynamic lint" tool to find tests with unknown sections is
something we can quite easily do, but for obvious reasons we can't block
any PR on it (even limiting it to a per-file basis, you still want to be
able to change obvious badness in a test without updating the link).

So what I believe we have rough agreement on:

   - Unversioned directories, probably under css/ to make the lint rules
   simpler and easier to understand for contributors.
   - Keep requiring <link rel=help> for CSS WG stuff.
   - Require versioned spec links.
   - Don't require links to specific spec sections, but encourage them (but
   don't block on it indefinitely!).

I think the only actionable items for us here are to move everything, to
change the lint to require versioned spec links, and to change the build
system to include unknown anchors (I think this is literally just deleting
an if block); and then the harness changes actionable by Peter?

/Geoffrey.