Re: Adding the testharness.js subrepo to the csswg-test repo

On Oct 17, 2013, at 3:51 AM, James Graham <james@hoppipolla.co.uk> wrote:

> On 16/10/13 21:30, Peter Linss wrote:
>>>> In other words, we wouldn't get the automatic updating of the
>>>> resources directory, as we do know by leaving them separate. Is this
>>>> what's really wanted?
>>> 
>>> Yes, for consistency with the main repo.
>> 
>> I have no issue with keeping the csswg repo consistent with the main
>> repo. But then we need to know which version of the resources repo the
>> main is using to keep in sync.
> 
> Not really. You need to upgrade when you have tested that the changes don't accidentially break your tests. I don't think anyone should have a requirement that the two version are in perfect sync.

I don't see having them in sync as a hard requirement, but it would be surprising to a test author if a test worked in the wpt repo and broke in the csswg repo (or vice versa) because the testharness.js files were different versions. If we start relying on different versions of resources, then it's just going to make it that much harder to combine the repos or move tests in the future.

> 
>> Is there a system for notifications when this changes? or perhaps a tag
>> in the resources repo that we can automatically sync to?
>> 
>> We also serve a copy of the resources repo at
>> http[s]://test.csswg.org/resources <http://test.csswg.org/resources> so
>> that references in the tests work. I also need to be able to keep that
>> in sync with the version referred to in WPT rather than the tip.
> 
> You need to keep that in sync with the version in your repository. That isn't really a harder problem than generally keeping any checkout up to date.

Syncing what I serve on test.csswg.org/resources to what's in the csswg repo is trivial, I was looking for some way of reasonably keeping that in sync with what the wpt repo is using. Even if it's just a note on the mailing list that it got updated... (and why)

> 
>>>> I haven't seen any discussion about making the resources directory a
>>>> submodule in the main WPT repo, so I'm not sure what the goals of
>>>> this change are.
>>> 
>>> It's been a submodule on the main repo for a while now.
>> 
>> That doesn't answer my question.
>> 
>> Was this done only so that people working on local clones of WPT (sort
>> of) automatically get the resources directory? or is there something
>> more to it?
>> 
>> e.g.: Has there also been a change in the way testharness.js gets used?
>> i.e. is it sill always '/resources/testharness.js' or are relative urls
>> now allowed too? Is the expectation now that the resources directory is
>> included in a packaged test suite? Remember that not all test suites are
>> simple clones of the WPT repo...
> 
> I don't understand what you mean by this.
> 
> Going forward, the supported configuration for running the web platform tests tests will be by using the provided servers and provided python script to initalise a full test environment including the required HTTP and websockets servers (in the future we can imagine also initalising additional servers such as FTP if we have tests that depend on them). Cloning the repository and all submodules will get you this full testing environment.
> 
> If you are a vendor trying to run the tests in automation it is expected that you will take the same approach to running them as just described, even if you also run other tests that require a different environment. Such other tests can be run seperately in whatever configuration they require.

Right, and that's a fine approach for someone doing general testing of the entire platform. But, as I keep bringing up, WGs are interested in running just a single test suite for a single spec for measuring and validating CR exit criteria. Furthermore, CSS test suites are the output of a build process, not the contents of the test repo.

Single spec CSS test suites still rely on having a /resources directory available, and historically that hasn't been part of the test suite. We could easily package a copy of the resources directory into a built test suite, but that suite won't necessarily get served at the root of a URL space, so references to /resources/testharness.js wouldn't refer to the copy in the test suite. I suppose it's something I'll just have to deal with in the build process.

Peter