Re: Unifying testsuite policy and getting rid of CSS exceptions

On Thu, Sep 14, 2017 at 7:23 PM, Geoffrey Sneddon <me@gsnedders.com> wrote:
> (BCC'd to www-style, to keep people in the loop, but can we keep all
> replies on the one list?)
>
> There was some discussion in the CSS WG a fortnight ago (minutes:
> http://www.w3.org/mid/CADhPm3t6F0RGzPsz=sG4JoCyPk6XEX+FJ+kzHyjHjkF_Q_H4tw@mail.gmail.com
> under the "Test Metadata" heading) about where new test suites should
> go.
>
> When the CSS testsuite was merged into wpt, we did it with the entire
> old repository just being put in `css/`, and adding a few extra lints
> (all of these to keep the build system working for the sake of keeping
> https://test.csswg.org/harness/ working, a requirement of the CSS WG
> being okay with the merge; these are a unique filename constraint, a
> requirement for <link rel=help> links, support files being in a
> support directory, though these don't suffice for all the requirements
> the build system).
>
> At that time, we had a few CSS test suites at the top level of WPT:
> cssom, cssom-view, and a few Houdini specs. Shortly after the merge,
> some people (including, I believe, the editor) decided to merge the
> entire cssom and cssom-view test suites into the top level
> directories, rather than having them split across multiple places.
> Following this, the code that runs https://test.csswg.org/harness/ was
> updated to build the CSS testsuite from the root of wpt, instead of
> just the css/ directory (though the lints only apply to css/).
>
> Beyond the extra lints, there's one other major difference: the
> directories in css/ are versioned (i.e., the directories map to the
> versioned short names), which has typically been an anti-goal given
> implementors rarely look at anything except the latest ED. However,
> given the general policy of not needlessly moving tests, we've mostly
> avoided doing that, so we now have some level 4 tests in level 3
> folders. Note the CSS build system currently only cares about <link
> rel=help> links, so directory names are irrelevant.
>
> The question is where we go from here, and there are three sides to this.
>
> 1. What we do with existing tests:
>
> a. We keep them where they are now.
>
> b. We move all CSS WG spec test suites into css/.
>
> c. We move all CSS WG spec test suites to the top-level.
>
>
> 1. Do we remove the versioning from future CSS WG test suites:
>
> a. Yes.
>
> b. No.
>
>
> 2. Where do new test suites go:
>
> a. At the top level.
>
> b. In css/.
>
> c. Up to the editor of the individual spec.
>
>
> 3. Do we move existing tests to match the above?
>
> a. Yes.
>
> b. No.
>
>
> 4. Where do we run the extra CSS lints:
>
> a. Only in css/.
>
> b. In all directories containing tests for CSS WG specs.
>
>
> As far as I'm aware, the status-quo is 1a, 2a, 3b, 4a. The CSS WG's
> preference is 1b, 2b, 3c, and I assume 4b (given without it the CSS
> WG's tooling is likely to get pretty broken pretty fast).
>
> There had been some talk about dropping the requirement for <link
> rel=help>, but this requires versioned folders (as you need to require
> somewhere has the version) for the build system's metadata
> requirements, and it would rely on folders within the spec mapping to
> anchors within the spec (and anchors aren't stable across versions,
> because specs get refactored).
>
> Does anyone have any preferences?
>
> /Geoffrey

Hello CSS WG!

The conclusion, from the discussion on public-test-infra and others
around WPT, is that from the WPT project there is a preference for:

 * Unversioned directories, probably under css/ to make the lint rules
simpler and easier to understand for contributors.
 * Keep requiring <link rel=help> for CSS WG stuff.
 * Require versioned spec links.
 * Don't require links to specific spec sections, but encourage them
(but don't block on it indefinitely!).

Peter agreed on IRC to update the build system and test harness so
that tests in no section and in unknown sections still appear (the
status quo is that they don't appear anywhere in any of the CSS WG's
tooling, which is awkward when test anchors change!).

Is the WG happy with this?

/Geoffrey

Received on Tuesday, 3 October 2017 15:26:26 UTC