Re: Unifying testsuite policy and getting rid of CSS exceptions

On Mon, Sep 25, 2017 at 1:14 PM Geoffrey Sneddon <me@gsnedders.com> wrote:

>
>
> On Mon, Sep 25, 2017 at 7:38 PM, Philip Jägenstedt <foolip@google.com>
> wrote:
>
>> On Mon, Sep 25, 2017 at 12:08 PM Geoffrey Sneddon <me@gsnedders.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Sep 25, 2017 at 6:31 PM, Philip Jägenstedt <foolip@google.com>
>>> wrote:
>>>
>>>> On Sun, Sep 24, 2017 at 3:48 AM Geoffrey Sneddon <me@gsnedders.com>
>>>> wrote:
>>>>
>>>>> On Sun, Sep 24, 2017 at 12:23 AM, Philip Jägenstedt <foolip@google.com
>>>>> > wrote:
>>>>>
>>>>>> On Sat, Sep 23, 2017 at 11:34 PM Geoffrey Sneddon <me@gsnedders.com>
>>>>>> wrote:
>>>>>>
>>>>>>> That said, the problems around the anchors could also be resolved by
>>>>>>> the CSS WG amending their tooling to put unknown anchors in a new "unknown"
>>>>>>> section, but I believe this would involve changes both in the build system
>>>>>>> and in the test harness, both of which are as far as I'm aware relatively
>>>>>>> unmaintained. Equally, we could treat unversioned URLs as the latest known
>>>>>>> version, but that again involves changes in both. (There appears to be
>>>>>>> separate code to group tests into spec sections in both the build system
>>>>>>> for the index pages and in the test harness for its results display, rather
>>>>>>> than the latter relying on the build system for the grouping.)
>>>>>>>
>>>>>>
>>>>>> Where is http://test.csswg.org/harness/ maintained? It ought to be a
>>>>>> 1- or 2-liner fix that we could do at the same time as removing version
>>>>>> numbers from directories.
>>>>>>
>>>>>
>>>>> It's all in http://hg.csswg.org/dev/harness/. It's also far from a
>>>>> 1-or-2-liner fix.
>>>>>
>>>>>
>>>>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/pages/ResultsPage.php#l433
>>>>> is the code that generates the results table, and the codepaths using
>>>>> writeSectionRows are the harder ones to fix (and the default codepath).
>>>>>
>>>>> You also need to change
>>>>> https://github.com/w3c/csswg-w3ctestlib/blob/master/Indexer.py to put
>>>>> unknown anchors somewhere in the index in the first place and then change
>>>>> http://hg.csswg.org/dev/harness/file/3e982ede4ca5/python/SynchronizeSpecLinks.py
>>>>> so that these reach the database the test harness works from.
>>>>>
>>>>> And I've probably missed something, because that's all I found looking
>>>>> last night without looking overly hard.
>>>>>
>>>>
>>>> Alan, Florian, other CSS WG folks, is there anyone that understands
>>>> this code still? It seems to me that even if we try very hard to not break
>>>> what's currently working, then over time it would break as sections change
>>>> names, or as typos occur. Without a tool that can automatically check the
>>>> whole test suite for being in the correct shape, it will be hard to keep it
>>>> from bitrotting.
>>>>
>>>> On the other hand, if such a tool did exist, then I think there's a
>>>> pretty easy fix here: make a best effort to keep things in good shape with
>>>> the line, but run the tool itself when it's needed to take a spec to REC
>>>> and fix the problems it reports then.
>>>>
>>>
>>> Having spoken to him on IRC, as I understand it Peter (Linss) is willing
>>> to amend it and the build system to create an "unknown" section in both the
>>> build output and the harness (and he's the one who's touched both most
>>> since the CSS WG adopted the system from whichever group had originally
>>> created it).
>>>
>>> Adding a "dynamic lint" tool to find tests with unknown sections is
>>> something we can quite easily do, but for obvious reasons we can't block
>>> any PR on it (even limiting it to a per-file basis, you still want to be
>>> able to change obvious badness in a test without updating the link).
>>>
>>> So what I believe we have rough agreement on:
>>>
>>>    - Unversioned directories, probably under css/ to make the lint
>>>    rules simpler and easier to understand for contributors.
>>>
>>>
>>>    - Keep requiring <link rel=help> for CSS WG stuff.
>>>
>>>
>>>    - Require versioned spec links.
>>>    - Don't require links to specific spec sections, but encourage them
>>>    (but don't block on it indefinitely!).
>>>
>>> I think the only actionable items for us here are to move everything, to
>>> change the lint to require versioned spec links, and to change the build
>>> system to include unknown anchors (I think this is literally just deleting
>>> an if block); and then the harness changes actionable by Peter?
>>>
>>
>> Awesome, we should do it!
>>
>> About the location, I guess we're sticking with css/ so that we don't
>> have to tackle the question of css/vendor-imports/ right now?
>>
>
> Mostly because I feel strongly that it's easier to tell contributors "if
> you're dealing with the css directory, there's these extra lints and extra
> rules" v. "if you're dealing with a directory that maps to a CSS spec,
> there's these extra lints and extra rules" (esp. now some former FXTF stuff
> is now in the CSS WG).
>

Sure, that works for me.

In the interest of moving forward, can you prepare a renaming script that
people can look at and run to understand what the change will be?