Re: Test case review

On Tue, 10 May 2011, Linss, Peter wrote:

>
> On May 10, 2011, at 9:15 AM, Linss, Peter wrote:
>
>> The CSS test repository contains files other than tests as well. Our test harness is not in the test repository, but we do have:
>> * files referenced from tests, ie: images and stylesheets
>> * reference pages (and files referenced from reference pages)
>> * scripts that generate tests
>> * apache server configuration files
>> * the test suite build code
>>
>> So yes, Shepherd will be managing and providing a review system for 
>> files other than tests as well. It will do more with a test file, like 
>> handle the test metadata, but it will have to handle other files as 
>> well.
>
> Another thing Shepherd has to manage is the relationship between tests 
> and their supporting files.
>
> For example, the way we organize our repository is that we have separate 
> directories for tests in development, submitted for review, and approved 
> tests. When a test is approved, it gets moved from the submitted 
> directory to the approved directory (this allows workflows by 
> interacting directly with the repository), when this happens, we need to 
> make sure all the supporting files get moved into the appropriate place 
> as well.
>
> Another case is revision management, the harness tracks revision data 
> and stores results against the revision of the test. When tests change, 
> old results are automatically disregarded. The harness needs to know 
> when a linked stylesheet, image or reference page has changed, even if 
> the test that uses it didn't.
>
> For the most part, we're trying to do this by getting the relationship 
> data directly from the test (e.g. parsing the src of an <img> tag), but 
> there are cases when the test can't refer the the necessary support 
> file, like a fail image that only shows up if the browser is computing 
> relative paths wrong. So we need a mechanism for declaring and tracking 
> these relationships as well.

This is where I think our visions differ. I think that building a 
good, or even good-enough, review system is enough of a challenge on its 
own without adding a huge range of testcase management features on top of 
it. I expect managing tests to be a *hard* problem. For example tests can 
be identified by:
a URL
an pair of URLs
a URL + a name

Any of those URLs may have a query component and/or a fragment id. So 
there is no relationship between a particular testcase and a particular 
file; there are a kind of cloud of files that might be assosciated with a 
given test. Trying to work out what those files are is impossible in the 
general case; for example they can be included by document.write("some" + 
"resource") or similar. So it will be a huge amount of work to keep all 
the metadata up to date even if simple cases are handled automatically. 
Basically I think that if the test can work with incorrect metadata then 
it is highly likely that the metadata will be wrong. So I am opposed to 
systems that try to store lots of rarely-used metadata.

I don't think that W3C needs to be in the business of regression testing 
browsers, so I don't see why we care about automatically deleting old 
results. Indeed I am skeptical about W3C publishing results at 
all. When W3C does publish results it should clearly be for a single 
revision of the browsers and a single revision of the testsuite. If newer 
results are avaliable the old ones could be discarded wholesale or 
archived somewhere well out of the way. It is the browser vendor's job to 
do their own day-to-day regression tracking and I don't think we need to 
be involved.

For approved/submitted tests, I am increasingly of the opinion that using 
different directories is likely the wrong approach. Using two branches in 
the VCS combined with a commit-based review system seems like it has a 
number of advantages. One could still have a w3-test.org show submitted/ 
and approved/ directories but in order to move tests from one to the 
other, one would simply apply commits from the submitted branch onto the 
approved branch, rather than trying to copy files and run the risk of 
leaving stuff behind or copying the wrong bits.

Received on Tuesday, 10 May 2011 20:20:52 UTC