Re: Unifying testsuite policy and getting rid of CSS exceptions

On Mon, Sep 25, 2017 at 7:38 PM, Philip Jägenstedt <foolip@google.com>
wrote:

> On Mon, Sep 25, 2017 at 12:08 PM Geoffrey Sneddon <me@gsnedders.com>
> wrote:
>
>>
>>
>> On Mon, Sep 25, 2017 at 6:31 PM, Philip Jägenstedt <foolip@google.com>
>> wrote:
>>
>>> On Sun, Sep 24, 2017 at 3:48 AM Geoffrey Sneddon <me@gsnedders.com>
>>> wrote:
>>>
>>>> On Sun, Sep 24, 2017 at 12:23 AM, Philip Jägenstedt <foolip@google.com>
>>>> wrote:
>>>>
>>>>> On Sat, Sep 23, 2017 at 11:34 PM Geoffrey Sneddon <me@gsnedders.com>
>>>>> wrote:
>>>>>
>>>>>> That said, the problems around the anchors could also be resolved by
>>>>>> the CSS WG amending their tooling to put unknown anchors in a new "unknown"
>>>>>> section, but I believe this would involve changes both in the build system
>>>>>> and in the test harness, both of which are as far as I'm aware relatively
>>>>>> unmaintained. Equally, we could treat unversioned URLs as the latest known
>>>>>> version, but that again involves changes in both. (There appears to be
>>>>>> separate code to group tests into spec sections in both the build system
>>>>>> for the index pages and in the test harness for its results display, rather
>>>>>> than the latter relying on the build system for the grouping.)
>>>>>>
>>>>>
>>>>> Where is http://test.csswg.org/harness/ maintained? It ought to be a
>>>>> 1- or 2-liner fix that we could do at the same time as removing version
>>>>> numbers from directories.
>>>>>
>>>>
>>>> It's all in http://hg.csswg.org/dev/harness/. It's also far from a
>>>> 1-or-2-liner fix.
>>>>
>>>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/
>>>> pages/ResultsPage.php#l433 is the code that generates the results
>>>> table, and the codepaths using writeSectionRows are the harder ones to fix
>>>> (and the default codepath).
>>>>
>>>> You also need to change https://github.com/w3c/
>>>> csswg-w3ctestlib/blob/master/Indexer.py to put unknown anchors
>>>> somewhere in the index in the first place and then change
>>>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/
>>>> python/SynchronizeSpecLinks.py so that these reach the database the
>>>> test harness works from.
>>>>
>>>> And I've probably missed something, because that's all I found looking
>>>> last night without looking overly hard.
>>>>
>>>
>>> Alan, Florian, other CSS WG folks, is there anyone that understands this
>>> code still? It seems to me that even if we try very hard to not break
>>> what's currently working, then over time it would break as sections change
>>> names, or as typos occur. Without a tool that can automatically check the
>>> whole test suite for being in the correct shape, it will be hard to keep it
>>> from bitrotting.
>>>
>>> On the other hand, if such a tool did exist, then I think there's a
>>> pretty easy fix here: make a best effort to keep things in good shape with
>>> the line, but run the tool itself when it's needed to take a spec to REC
>>> and fix the problems it reports then.
>>>
>>
>> Having spoken to him on IRC, as I understand it Peter (Linss) is willing
>> to amend it and the build system to create an "unknown" section in both the
>> build output and the harness (and he's the one who's touched both most
>> since the CSS WG adopted the system from whichever group had originally
>> created it).
>>
>> Adding a "dynamic lint" tool to find tests with unknown sections is
>> something we can quite easily do, but for obvious reasons we can't block
>> any PR on it (even limiting it to a per-file basis, you still want to be
>> able to change obvious badness in a test without updating the link).
>>
>> So what I believe we have rough agreement on:
>>
>>    - Unversioned directories, probably under css/ to make the lint rules
>>    simpler and easier to understand for contributors.
>>
>>
>>    - Keep requiring <link rel=help> for CSS WG stuff.
>>
>>
>>    - Require versioned spec links.
>>    - Don't require links to specific spec sections, but encourage them
>>    (but don't block on it indefinitely!).
>>
>> I think the only actionable items for us here are to move everything, to
>> change the lint to require versioned spec links, and to change the build
>> system to include unknown anchors (I think this is literally just deleting
>> an if block); and then the harness changes actionable by Peter?
>>
>
> Awesome, we should do it!
>
> About the location, I guess we're sticking with css/ so that we don't have
> to tackle the question of css/vendor-imports/ right now?
>

Mostly because I feel strongly that it's easier to tell contributors "if
you're dealing with the css directory, there's these extra lints and extra
rules" v. "if you're dealing with a directory that maps to a CSS spec,
there's these extra lints and extra rules" (esp. now some former FXTF stuff
is now in the CSS WG).

/g