Re: Unifying testsuite policy and getting rid of CSS exceptions

On Sun, Sep 24, 2017 at 3:48 AM Geoffrey Sneddon <me@gsnedders.com> wrote:

> On Sun, Sep 24, 2017 at 12:23 AM, Philip Jägenstedt <foolip@google.com>
> wrote:
>
>> On Sat, Sep 23, 2017 at 11:34 PM Geoffrey Sneddon <me@gsnedders.com>
>> wrote:
>>
>>> That said, the problems around the anchors could also be resolved by the
>>> CSS WG amending their tooling to put unknown anchors in a new "unknown"
>>> section, but I believe this would involve changes both in the build system
>>> and in the test harness, both of which are as far as I'm aware relatively
>>> unmaintained. Equally, we could treat unversioned URLs as the latest known
>>> version, but that again involves changes in both. (There appears to be
>>> separate code to group tests into spec sections in both the build system
>>> for the index pages and in the test harness for its results display, rather
>>> than the latter relying on the build system for the grouping.)
>>>
>>
>> Where is http://test.csswg.org/harness/ maintained? It ought to be a 1-
>> or 2-liner fix that we could do at the same time as removing version
>> numbers from directories.
>>
>
> It's all in http://hg.csswg.org/dev/harness/. It's also far from a
> 1-or-2-liner fix.
>
>
> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/pages/ResultsPage.php#l433
> is the code that generates the results table, and the codepaths using
> writeSectionRows are the harder ones to fix (and the default codepath).
>
> You also need to change
> https://github.com/w3c/csswg-w3ctestlib/blob/master/Indexer.py to put
> unknown anchors somewhere in the index in the first place and then change
> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/python/SynchronizeSpecLinks.py
> so that these reach the database the test harness works from.
>
> And I've probably missed something, because that's all I found looking
> last night without looking overly hard.
>

Alan, Florian, other CSS WG folks, is there anyone that understands this
code still? It seems to me that even if we try very hard to not break
what's currently working, then over time it would break as sections change
names, or as typos occur. Without a tool that can automatically check the
whole test suite for being in the correct shape, it will be hard to keep it
from bitrotting.

On the other hand, if such a tool did exist, then I think there's a pretty
easy fix here: make a best effort to keep things in good shape with the
line, but run the tool itself when it's needed to take a spec to REC and
fix the problems it reports then.