CSS tests in web-platform-tests

I see there was some discussion about merging the CSS testsuite with
web-platform-tests happened in the CSS F2F meeting. It was a bit hard to
tell from the minutes if any concrete conclusions were reached, but I
think this is an important piece of work and would like to set out a
strawman proposal for how to proceed.

I have seen various options for integrating the testsuites put forward.
In particular the option of using submodules has come up before. I think
this is the wrong approach for several reasons:

* Submodules aren't really designed for situation where you want to
remain in lock-step with upstream, because every time you want to update
upstream you have to update the submodule pointer. With some effort,
this could be automated on the upstream side but it would mean that
every time someone pulled the testsuite they would have to make sure
they also updated submodules. This is very easy to get wrong.

* Having the CSS tests and other tests in seperate repos pushes the
burden of effort onto the people with least understanding of the test
setup; authors. Particularly for first-time authors it's not at all
obvious why different testsuites should live in different repositories
and we spend a lot of effort on testthewebforward.org explaining how to
deal with this separation.

I think a much better plan is to reverse the flow of tests; instead of
pulling tests from a CSS repo into web-platform-tests, we put all the
tests in web-platform-tests and then, to support existing CSSWG code
that assumes a particular repository setup, we export CSS-related tests
into a repository in which all the existing tools work. This requires a
little work but once set up will make things much easier for everyone.

If CSS tools are flexible enough to work on the web-platform-tests
repository directly that would obviously be an even better idea, but I
conservatively assume that they are not. If I am wrong, you can skip the
next two paragraphs.

There are a couple of possible ways of organizing such a setup. The
simplest, technically, is to put all the CSS tests in a single
subfolder. Then they can be exported to their own repository using
git-subtree. However, based on the proposed layout of the CSS repo, it
would likely still be necessary to rewrite the history of this
repository e.g. to get the tests under a tests/ directory in the CSS
repository (git filter-branch can be used for this purpose). The major
difficulty with this scheme is that it makes CSS tests different to
other tests because the top directory name will not map to the TR
shortname of the spec. This will break some tools that depend on this
property.

A better idea then is to follow the existing conventions with CSS and
have the test directories at the top level. This leaves a certain amount
of work to do to export to a CSS-specific repository, but it is
technically possible. On approach would be to simply fetch the whole
history of web-platform-tests into the CSS repo and then merge the two
histories. As part of the merge commit we would delete unwanted files
(i.e. those not corresponding to CSS specs) and move CSS-related files
to the tests/ subdirectory. Similar effects can be achieved by using
filter-branch --subdirectory-filter or git subtree to extract the
history of each branch, and then fetch that into the CSS repo, before
moving it into place as part of a merge.

Once we have a repository that the CSS tools can operate on, there are a
few more obstacles that need to be overcome. I don't think that any of
these are particularly challenging:

* Build step. AIUI CSS still requires a build step for some tests.
web-platform-tests assumes that the repository contains the actual test
files in order to minimise the difficulty of running tests. I imagine
any problems here can be solved by checking in the built versions of the
tests and having a naming convention to prevent source files being
detected as tests, if that's a problem.

* Additional test requirements. CSS impose some additional constraints
on tests that are not shared by all tests in web-platform-tests. This
isn't a problem; from the point of view of CSS tests checked-in to
web-platform-tests should be regarded as a pool of submissions, with
tests only vetted by the WG being used for conformance testing. AIUI
this is very like the existing system where people can submit tests into
the repository but may be asked to make changes before they are marked
as approved.

* Code review. web-platform-tests has a separate review system which
does not try to implement all the things that CSS require. However when
a CSSWG member reviews a test and finds that it is acceptable to go into
the conformance suite it is annoying makework if they have to manually
review it in two places. I suggest we implement a system where adding a
comment to a merged github PR like "CSSWG:accepted" automatically sets a
test as accepted in the CSSWG system, pulling in whatever data is needed
from the existing review.

Received on Friday, 23 May 2014 14:31:30 UTC