Getting browser vendors running and submitting tests

[CC'd to public-test-infra; this is really mostly about CSS, can we avoid
cross-posting replies and keeping them on public-css-testsuite?]

Myself and fantasai chaired a session at TPAC earlier today on this
subject. The minutes are at <
http://www.w3.org/2015/10/28-testing-minutes.html> (thanks fantasai!).

Notably:

* The build system makes it difficult for vendors to upstream their fixes
and their own tests (as vendors can't just fix the files they run; they
have to find some other file, fix that, and then work out how to build the
test suite). As a result, they just fix the built files and don't upstream
the fixes and often avoid updating the copy of the test suite because it
would trample their fixes. We should get rid of the build system (at least
insofar as it modifies/moves files), as I don't think anyone likely to run
the tests doesn't support HTML at this point (yay the HTML5 parser!).

* We should get rid of metadata, at least in the common case, required to
submit tests to the the test suite, as this is an impediment to upstreaming
tests. (See also my recent thread on public-css-testsuite, "Simplifying
metadata".)

These would make it far easier for browsers to run the up-to-date CSS test
suite in their CI systems, to push fixes to the test suite, and to release
tests their have into the test suite.

/gsnedders

Received on Wednesday, 28 October 2015 08:22:01 UTC