Re: Towards a better testsuite: Build System

On 03/24/2016 01:00 PM, Geoffrey Sneddon wrote:
>
> The other significant complication when it comes to csswg-test is the
> build system. Because of the necessity of building the testsuite first
> it makes it more complicated to fix tests when they're failing; you
> have to know how to find the source file and then be able to build the
> testsuite after fixing it. Historically the build system existed to
> deal with the variety in what UAs support, whether they supported HTML
> and/or XHTML; this is a far smaller deal nowadays—of the UAs running
> the CSS testsuite or likely to do so in the future, the only one I'm
> aware of that doesn't support both well is Servo (and that's likely to
> change so can possibly be ignored here).

I don't have much opinion on whether we keep or discard the
build system, but I don't think in any case that it should
be necessary in order for people to run or otherwise use the
tests. Tests are identified by filename, and run just fine
without the build system, so there's no need to build in the
general case.

For results reporting, we have Shepherd, which tracks which
tests belong to which suites, and doesn't much care where
they end up. The build system isn't necessary here, either.

However, individual vendors may need scripts to convert the
test-reference linkages into their preferred format E.g.
for Mozilla, we do need to generate reftest manifest files,
which are currently constructed by the build system. But
that can be done with a lighter-weight system that just
generates manifests in place per directory.

(As for adopting a "filename convention" for mapping the
tests and references... No. There are thousands of CSS tests
that use the same reference file. Whoever wants a "filename
convention" can make 1000 copies of each common reference if
they want, but I refuse to support such nonsense in the CSSWG
repository.)

~fantasai

Received on Friday, 8 April 2016 17:00:58 UTC