- From: James Graham <james@hoppipolla.co.uk>
- Date: Fri, 23 May 2014 15:31:04 +0100
- To: "public-test-infra@w3.org" <public-test-infra@w3.org>
I see there was some discussion about merging the CSS testsuite with web-platform-tests happened in the CSS F2F meeting. It was a bit hard to tell from the minutes if any concrete conclusions were reached, but I think this is an important piece of work and would like to set out a strawman proposal for how to proceed. I have seen various options for integrating the testsuites put forward. In particular the option of using submodules has come up before. I think this is the wrong approach for several reasons: * Submodules aren't really designed for situation where you want to remain in lock-step with upstream, because every time you want to update upstream you have to update the submodule pointer. With some effort, this could be automated on the upstream side but it would mean that every time someone pulled the testsuite they would have to make sure they also updated submodules. This is very easy to get wrong. * Having the CSS tests and other tests in seperate repos pushes the burden of effort onto the people with least understanding of the test setup; authors. Particularly for first-time authors it's not at all obvious why different testsuites should live in different repositories and we spend a lot of effort on testthewebforward.org explaining how to deal with this separation. I think a much better plan is to reverse the flow of tests; instead of pulling tests from a CSS repo into web-platform-tests, we put all the tests in web-platform-tests and then, to support existing CSSWG code that assumes a particular repository setup, we export CSS-related tests into a repository in which all the existing tools work. This requires a little work but once set up will make things much easier for everyone. If CSS tools are flexible enough to work on the web-platform-tests repository directly that would obviously be an even better idea, but I conservatively assume that they are not. If I am wrong, you can skip the next two paragraphs. There are a couple of possible ways of organizing such a setup. The simplest, technically, is to put all the CSS tests in a single subfolder. Then they can be exported to their own repository using git-subtree. However, based on the proposed layout of the CSS repo, it would likely still be necessary to rewrite the history of this repository e.g. to get the tests under a tests/ directory in the CSS repository (git filter-branch can be used for this purpose). The major difficulty with this scheme is that it makes CSS tests different to other tests because the top directory name will not map to the TR shortname of the spec. This will break some tools that depend on this property. A better idea then is to follow the existing conventions with CSS and have the test directories at the top level. This leaves a certain amount of work to do to export to a CSS-specific repository, but it is technically possible. On approach would be to simply fetch the whole history of web-platform-tests into the CSS repo and then merge the two histories. As part of the merge commit we would delete unwanted files (i.e. those not corresponding to CSS specs) and move CSS-related files to the tests/ subdirectory. Similar effects can be achieved by using filter-branch --subdirectory-filter or git subtree to extract the history of each branch, and then fetch that into the CSS repo, before moving it into place as part of a merge. Once we have a repository that the CSS tools can operate on, there are a few more obstacles that need to be overcome. I don't think that any of these are particularly challenging: * Build step. AIUI CSS still requires a build step for some tests. web-platform-tests assumes that the repository contains the actual test files in order to minimise the difficulty of running tests. I imagine any problems here can be solved by checking in the built versions of the tests and having a naming convention to prevent source files being detected as tests, if that's a problem. * Additional test requirements. CSS impose some additional constraints on tests that are not shared by all tests in web-platform-tests. This isn't a problem; from the point of view of CSS tests checked-in to web-platform-tests should be regarded as a pool of submissions, with tests only vetted by the WG being used for conformance testing. AIUI this is very like the existing system where people can submit tests into the repository but may be asked to make changes before they are marked as approved. * Code review. web-platform-tests has a separate review system which does not try to implement all the things that CSS require. However when a CSSWG member reviews a test and finds that it is acceptable to go into the conformance suite it is annoying makework if they have to manually review it in two places. I suggest we implement a system where adding a comment to a merged github PR like "CSSWG:accepted" automatically sets a test as accepted in the CSSWG system, pulling in whatever data is needed from the existing review.
Received on Friday, 23 May 2014 14:31:30 UTC