Re: Unifying testsuite policy and getting rid of CSS exceptions

On Fri, Sep 22, 2017 at 12:12 AM James Graham <james@hoppipolla.co.uk>
wrote:

> On 21/09/17 03:22, Alan Stearns wrote:
> > On 9/20/17, 7:02 PM, "fantasai" <fantasai.lists@inkedblade.net> wrote:
> >
> >      On 09/15/2017 06:54 AM, James Graham wrote:
> >      > In practice anyone sufficiently knowledgeable to be interacting
> >      > with a test can likely identify the relevant parts of the specs
> >      > rather quickly.
> >
> >      As a CSS spec editor who was tasked with reviewing some totally
> >      uncommented layout tests, I disagree with this statement.
> >
> > And here’s a recent non-CSS example:
> http://logs.glob.uno/?c=freenode%23whatwg&s=29+Aug+2017&e=29+Aug+2017&h=comment#c1036038
> >
> > I expect wanderview is sufficiently knowledgeable, but could have
> benefitted from some test comments/metadata there.
>
> I;m obviously not claiming that comments have no value. But a link to
> the spec is not a comment, it's less precise than a well-written comment
> to explain the details of what the test author believed the test was
> testing. Nevertheless I agree that it has some value. But requiring
> links to pass the link also has a cost in terms of developers'
> willingness to contribute tests. Given the relative lack of vendor
> contributions to CSS tests I claim that reducing the barrier to entry is
> a more significant win to the platform as a whole than the value offered
> by a mandatory spec link.
>

I would also like to remove the <link rel=help> requirement at some point,
but I would be surprised if that's currently the main impediment for
Chromium developers. My guesses are rather:

   - We're still not importing all of the tests.
   - The mix of versioned and unversioned directories in css/ and in the
   root. Most ridiculous is css-cascade + css/css-cascade-3 +
   css/css-cascade-4 with a total of just 11 tests.
   - Significant duplication between wpt and LayoutTests, as some test
   collections have a common origin.
   - All reftests fail with ERROR on Chrome
   <https://github.com/w3c/wptdashboard/issues/97> in wpt.fyi, anyone who's
   noticed the failures without investigating would think the tests are just
   hopelessly broken.


On "unification", here's what I summarized as a compromise in a private
exchange with Florian:

   - Unversioned directories. Under css/ to make the lint rules simpler, or
   in the root with a few more lines of lint logic, doesn't matter.
   - Keep requiring <link rel=help> for CSS WG stuff.
   - Don't require links to specific spec sections if upstreaming new tests
   in bulk, because then it just won't happen.
   - Encourage versioned spec links in documentation, but don't require it,
   it's easy to fix in bulk on spec level bump.


It's not the end state I'd like, but it addresses what I think are the
biggest problems.

The root of the tension is that some of us (including me) think that spec
levels (aka long-lived stabilization branches) are a bit silly, but a
proper fix for that has to come from within the CSS WG I think. Maybe we
could revisit again in 3-6 months to see how things have worked in the CSS
WG process?