Re: Syncing upstream changes in testsuites

On Thu, 21 Mar 2013, Robin Berjon wrote:

> On 21/03/2013 16:36 , James Graham wrote:
>> On Thu, 21 Mar 2013, Robin Berjon wrote:
>>> Just a silly thought, I may be missing something, but does that sync
>>> really need to use git? I mean presumably it's a read-only sync that
>>> only needs to happen once a day or so (it's not a big deal if runs are
>>> 24h behind the repo I would think). In that case, if git is an issue,
>>> one can just grab and unpack:
>>>
>>>    https://github.com/w3c/web-platform-tests/archive/master.zip
>> 
>> That works iff you don't have any local patches to the testsuite. That
>> typically won't be the case, almost certainly for testharnessreport.js
>> which is explicitly intended to be patched, but potentially also for
>> other files (doubly so if you have a single web-platform-tests
>> repository that contains both the upstream tests and local tests you
>> have written but not yet submitted to W3C). The operation you want to
>> run to sync the tests is a lot more like "git pull --rebase" than
>> "delete and reclone".
>
> Are you really sure that that's a requirement? Overwriting 
> testharnessreport.js might be something, but that's just a file copy 
> operation — it strikes me as being within reach.

No. I'm sure it was a requirement when Opera imported external tests in 
which we didn't have any control over the upstream and there wasn't any 
testharnessreport.js style integration point. Removing one facet of that 
problem is more or less why we have testharnessreport.js. It might still 
be a requirement if there are particular tests that interact badly with a 
particular vendor's infrastructure but are otherwise OK, for example.

> If you work with rebase and local changes, then it's a given that things will 
> fail at some point. There's bound to be a conflict at some point.

Yes, it does mean that importing new tests can't be 100% automated.

> Were I integrating these tests into something I run regularly such as 
> continuous integration, I would run off the vanilla source (with possibly a 
> few forced overwrites like the one mentioned above), but I wouldn't mix in my 
> new stuff. I'd keep that in a separate directory or some such, and I'd run 
> those separately.

That could work I guess, although I'm not sure how well it would go in 
practice to have two different copies of the repository with some tests in 
one, some tests in the other and tests moving between them as they get 
approved. Certainly it seems undesirable if you are trying to record test 
results because it means that the URL to tests changes over time.