Re: Unifying testsuite policy and getting rid of CSS exceptions

On Wed, Sep 27, 2017 at 5:09 PM James Graham <james@hoppipolla.co.uk> wrote:

> On 25/09/17 11:08, Geoffrey Sneddon wrote:
>
> >     - Unversioned directories, probably under css/ to make the lint rules
> >     simpler and easier to understand for contributors.
> >     - Keep requiring <link rel=help> for CSS WG stuff.
> >     - Require versioned spec links.
> >     - Don't require links to specific spec sections, but encourage them
> (but
> >     don't block on it indefinitely!).
>
> FTR it seems like this consensus is just the result of everyone giving a
> little without changing their fundamental position, and doesn't
> represent a solution that fulfills anyone's goals. It doesn't make
> authoring tests much easier because you still need to learn about
> special rules under css/ and take the time to comply with them. In
> particular t still means that a single-vendor reftest is always easier
> to write than a cross-browser one even when the actual test is
> identical. And it doesn't really fulfill the CSSWG goals of
> documentation or outsourcing the burden of creating an IR to test
> authors because manual work will be required to deal with "unknown"
> tests to ensure that they are in the same level at transition time as
> they were when written.
>
> I'm happy to go ahead with this since there is agreement, but I suspect
> it will turn out to be an unsatisfactory arrangement.
>

I also doubt that this is the final state of things, but I have to say it
does fulfill the most important of my goals, which is to make the
organization more approachable (one directory per spec), allowing Blink's
layout and style teams to work more effectively with them.

Having to add <link rel=help> even where it's obvious from context is a bit
annoying, but it seems quite tolerable as long as it shows up early in the
process (presubmit or review) when people are already expecting feedback.
It may well turn out in 3-6 months that this is the top complaint about
working with wpt, and then I think we should address it.

I also expect that wpt.fyi will keep getting better to the point where
using that is the only sensible choice, and then we might have this
discussion again, about how to deal with what is effectively long lived
branches of the CSS specs, but not of the test suite.

Progress?