Re: Importing W3C's test suite into implementor's test suites.

On Thu, 21 Mar 2013, Tobie Langel wrote:

> On Thursday, March 21, 2013 at 10:55 PM, James Graham wrote:
>> FWIW the model at Opera is that tests are written and reviewed internally
>> and then submitted to W3C as a seperate step. This can involve some
>> changes to the tests e.g. to change internal server names to W3C server
>> names, although we have got better at either avoiding these problems or
>> mitigating them. I'm not claiming that this is the best model, but I
>> imagine there are a number of organisations that will not be submitting
>> tests as soon as they are written, but waiting for a variety of reasons.
>
> This is certainly a scenario we have to support.
>
> How do you handle syncing back the W3C tests?

Badly :)

The approach I really want to use is to have a single repository with the 
w3c as a remote and master as w3c/master + all unsubmitted tests and local 
patches. That way, once the tests are accepted by the W3C the test URL 
doesn't change. But this is still problematic as it mixes together all 
unsubmitted changes from any in-progress testsuites, so seperating things 
out into clean branches isn't easy (and isn't possible without changing 
the commit SHA1s). Keeping multiple internal branches seems cleaner, but 
still doesn't quite work because one needs to swith branches to run 
unsubmitted tests and to deal with patches that should be on all internal 
branches in a sane way.

I also don't know how well it works when you keep all 
the tests in the same repository as your source code. Perhaps OK if you 
can make the tests a submodule or similar, but again that depends on using 
git internally (not a problem for us, but a big problem for others).

> Do you remove yours?
>
> Do you not care about duplicates?

In practice we try to remove ours and just use the W3C copy. But it is a 
manual process carried out on a case-by-case basis and so is totally 
unreliable.

Received on Thursday, 21 March 2013 22:19:08 UTC