Re: Unifying testsuite policy and getting rid of CSS exceptions

On Mon, Sep 25, 2017 at 12:08 PM Geoffrey Sneddon <me@gsnedders.com> wrote:

>
>
> On Mon, Sep 25, 2017 at 6:31 PM, Philip Jägenstedt <foolip@google.com>
> wrote:
>
>> On Sun, Sep 24, 2017 at 3:48 AM Geoffrey Sneddon <me@gsnedders.com>
>> wrote:
>>
>>> On Sun, Sep 24, 2017 at 12:23 AM, Philip Jägenstedt <foolip@google.com>
>>> wrote:
>>>
>>>> On Sat, Sep 23, 2017 at 11:34 PM Geoffrey Sneddon <me@gsnedders.com>
>>>> wrote:
>>>>
>>>>> That said, the problems around the anchors could also be resolved by
>>>>> the CSS WG amending their tooling to put unknown anchors in a new "unknown"
>>>>> section, but I believe this would involve changes both in the build system
>>>>> and in the test harness, both of which are as far as I'm aware relatively
>>>>> unmaintained. Equally, we could treat unversioned URLs as the latest known
>>>>> version, but that again involves changes in both. (There appears to be
>>>>> separate code to group tests into spec sections in both the build system
>>>>> for the index pages and in the test harness for its results display, rather
>>>>> than the latter relying on the build system for the grouping.)
>>>>>
>>>>
>>>> Where is http://test.csswg.org/harness/ maintained? It ought to be a
>>>> 1- or 2-liner fix that we could do at the same time as removing version
>>>> numbers from directories.
>>>>
>>>
>>> It's all in http://hg.csswg.org/dev/harness/. It's also far from a
>>> 1-or-2-liner fix.
>>>
>>>
>>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/pages/ResultsPage.php#l433
>>> is the code that generates the results table, and the codepaths using
>>> writeSectionRows are the harder ones to fix (and the default codepath).
>>>
>>> You also need to change
>>> https://github.com/w3c/csswg-w3ctestlib/blob/master/Indexer.py to put
>>> unknown anchors somewhere in the index in the first place and then change
>>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/python/SynchronizeSpecLinks.py
>>> so that these reach the database the test harness works from.
>>>
>>> And I've probably missed something, because that's all I found looking
>>> last night without looking overly hard.
>>>
>>
>> Alan, Florian, other CSS WG folks, is there anyone that understands this
>> code still? It seems to me that even if we try very hard to not break
>> what's currently working, then over time it would break as sections change
>> names, or as typos occur. Without a tool that can automatically check the
>> whole test suite for being in the correct shape, it will be hard to keep it
>> from bitrotting.
>>
>> On the other hand, if such a tool did exist, then I think there's a
>> pretty easy fix here: make a best effort to keep things in good shape with
>> the line, but run the tool itself when it's needed to take a spec to REC
>> and fix the problems it reports then.
>>
>
> Having spoken to him on IRC, as I understand it Peter (Linss) is willing
> to amend it and the build system to create an "unknown" section in both the
> build output and the harness (and he's the one who's touched both most
> since the CSS WG adopted the system from whichever group had originally
> created it).
>
> Adding a "dynamic lint" tool to find tests with unknown sections is
> something we can quite easily do, but for obvious reasons we can't block
> any PR on it (even limiting it to a per-file basis, you still want to be
> able to change obvious badness in a test without updating the link).
>
> So what I believe we have rough agreement on:
>
>    - Unversioned directories, probably under css/ to make the lint rules
>    simpler and easier to understand for contributors.
>
>
>    - Keep requiring <link rel=help> for CSS WG stuff.
>
>
>    - Require versioned spec links.
>    - Don't require links to specific spec sections, but encourage them
>    (but don't block on it indefinitely!).
>
> I think the only actionable items for us here are to move everything, to
> change the lint to require versioned spec links, and to change the build
> system to include unknown anchors (I think this is literally just deleting
> an if block); and then the harness changes actionable by Peter?
>

Awesome, we should do it!

About the location, I guess we're sticking with css/ so that we don't have
to tackle the question of css/vendor-imports/ right now?