Re: Proposal to move WebApps' tests to github [Was: Re: Opera's SSE Tests on GitHub]

On 21/01/2013 15:34 , SULLIVAN, BRYAN L wrote:
> I support this move. It will make it easier for us to contribute to
> the test suites. But we don't have anyone to do the move of the
> existing tests, specifically. Not sure if that's what you meant by
> "who you expect to do the work".

Moving the tests may or may not be hard, it entirely depends on the 
current state of your TS (knowing that if you're going to move, you want 
to avoid moving something in a crappy state).

Here's the short of what we did for HTML (which covers tests for HTML, 
Canvas, and Microdata):

• Nuke the approved/submitted directory structure. Not only is that a 
bad setup for the many reasons that have been discussed already, but in 
the process I noticed that it was common for people to forget removing 
their submission after the tests were approved (leading to a lot of 
duplicate tests).

• Move all the approved tests to a root directory.

• Move all the tests so that they are in subdirectories that match the 
IDs of sections in the relevant spec. I guess that this step is optional 
for smaller specs, but it has the nice property that you get vernacular 
metadata to link back to the spec instead of relying on authors to get 
it right (which they often don't, there's a lot of cut'n'paste going on).

• For the submissions that remain, integrate them in branches and add 
them as pull requests. For HTML we had a huge backlog of submissions so 
that we haven't yet integrated everything, but in any case this is 
useful work as it provides a clear list of submissions in a format that 
is amenable to comments, changes, etc. with a paper trail (see 
https://github.com/w3c/html-testsuite/pulls and 
https://github.com/w3c/html-testsuite/pull/18 for examples).

• All future submissions should come as PRs.

• So far we're checking that submitters have filled out the agreement by 
hand (which is easy) but we're looking at using something like 
http://clahub.com/ to automate the process (I wouldn't block on this 
though, running it by hand is trivial).

We're in the process of improving all the tooling around this, which 
will be spread out over several months. But I certainly think that other 
groups would greatly benefit from sharing the same infrastructure.

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Monday, 21 January 2013 15:15:00 UTC