Re: Adding the testharness.js subrepo to the csswg-test repo

On 25/10/13 02:33, Peter Linss wrote:
>
> On Oct 17, 2013, at 3:51 AM, James Graham <james@hoppipolla.co.uk>
> wrote:
>
>> On 16/10/13 21:30, Peter Linss wrote:
>>>>> In other words, we wouldn't get the automatic updating of
>>>>> the resources directory, as we do know by leaving them
>>>>> separate. Is this what's really wanted?
>>>>
>>>> Yes, for consistency with the main repo.
>>>
>>> I have no issue with keeping the csswg repo consistent with the
>>> main repo. But then we need to know which version of the
>>> resources repo the main is using to keep in sync.
>>
>> Not really. You need to upgrade when you have tested that the
>> changes don't accidentially break your tests. I don't think anyone
>> should have a requirement that the two version are in perfect
>> sync.
>
> I don't see having them in sync as a hard requirement, but it would
> be surprising to a test author if a test worked in the wpt repo and
> broke in the csswg repo (or vice versa) because the testharness.js
> files were different versions. If we start relying on different
> versions of resources, then it's just going to make it that much
> harder to combine the repos or move tests in the future.

I think testharness.js is infrequently enough updated that this is not a 
problem. You can find out when it is updated by subscribing to the 
relevant github repo.

>>>>> I haven't seen any discussion about making the resources
>>>>> directory a submodule in the main WPT repo, so I'm not sure
>>>>> what the goals of this change are.
>>>>
>>>> It's been a submodule on the main repo for a while now.
>>>
>>> That doesn't answer my question.
>>>
>>> Was this done only so that people working on local clones of WPT
>>> (sort of) automatically get the resources directory? or is there
>>> something more to it?
>>>
>>> e.g.: Has there also been a change in the way testharness.js gets
>>> used? i.e. is it sill always '/resources/testharness.js' or are
>>> relative urls now allowed too? Is the expectation now that the
>>> resources directory is included in a packaged test suite?
>>> Remember that not all test suites are simple clones of the WPT
>>> repo...
>>
>> I don't understand what you mean by this.
>>
>> Going forward, the supported configuration for running the web
>> platform tests tests will be by using the provided servers and
>> provided python script to initalise a full test environment
>> including the required HTTP and websockets servers (in the future
>> we can imagine also initalising additional servers such as FTP if
>> we have tests that depend on them). Cloning the repository and all
>> submodules will get you this full testing environment.
>>
>> If you are a vendor trying to run the tests in automation it is
>> expected that you will take the same approach to running them as
>> just described, even if you also run other tests that require a
>> different environment. Such other tests can be run seperately in
>> whatever configuration they require.
>
> Right, and that's a fine approach for someone doing general testing
> of the entire platform. But, as I keep bringing up, WGs are
> interested in running just a single test suite for a single spec for
> measuring and validating CR exit criteria. Furthermore, CSS test
> suites are the output of a build process, not the contents of the
> test repo.

The right solution to this is to take the results from the runs that 
vendors are already doing. Of course this requires us to get to a point 
where the tests are being regularly run by vendors. But since that ought 
to be the most urgent priority of the testing effort anyway, it doesn't 
seem like a bad idea to focus attention on the issue.

Personally I would quite happily exclude any vendors who aren't running 
the W3C tests in a CI setup, and making the results public, from 
consideration in any process transition requirements. If people aren't 
doing this there is not even the weakest of guarantees that tests which 
pass today are going to continue to pass in the future. This rather 
undermines the point of having the tests as to demonstrate interoperability.

> Single spec CSS test suites still rely on having a /resources
> directory available, and historically that hasn't been part of the
> test suite. We could easily package a copy of the resources directory
> into a built test suite, but that suite won't necessarily get served
> at the root of a URL space, so references to
> /resources/testharness.js wouldn't refer to the copy in the test
> suite. I suppose it's something I'll just have to deal with in the
> build process.

I strongly recommend adopting the approach that web-platform-tests is 
taking where the full environment is bundled with the testsuite so that 
a single command will launch webservers on a set of free ports, as 
required for cross-origin testing, all of which have the testharness.js 
repo served under /resources. This neatly solves all the problems you 
are describing and provides an almost-unified user experience (the 
unified experience occurring once we only have a single test repo. of 
course).

Received on Friday, 25 October 2013 09:52:36 UTC