Re: Test case review

On Tue, 10 May 2011, Linss, Peter wrote:

(I cut too much here, but should say that if you could explain *why* the 
CSS WG has decided it needs these things it might help me understand your 
point of view)

>> I don't think that W3C needs to be in the business of regression testing
>> browsers, so I don't see why we care about automatically deleting old
>> results. Indeed I am skeptical about W3C publishing results at
>> all. When W3C does publish results it should clearly be for a single
>> revision of the browsers and a single revision of the testsuite. If newer
>> results are avaliable the old ones could be discarded wholesale or
>> archived somewhere well out of the way. It is the browser vendor's job to
>> do their own day-to-day regression tracking and I don't think we need to
>> be involved.
>
> You misunderstand here, it's not tracking the revisions of the browsers 
> (well, it does store the version of the browser, but it doesn't do 
> anything with that data). What I'm talking about are revisions of the 
> _tests_. When a test is updated, the harness ignores results stored 
> against previous versions of the test. We find issues with the tests all 
> the time and update them as needed.

No, I undersood. I still don't understand why we care. As far as I know 
the only use for saving old test results is regression tracking. Why do we 
need to delete the results of specific tests rather than discard the whole 
results set?

>> For approved/submitted tests, I am increasingly of the opinion that using
>> different directories is likely the wrong approach. Using two branches in
>> the VCS combined with a commit-based review system seems like it has a
>> number of advantages. One could still have a w3-test.org show submitted/
>> and approved/ directories but in order to move tests from one to the
>> other, one would simply apply commits from the submitted branch onto the
>> approved branch, rather than trying to copy files and run the risk of
>> leaving stuff behind or copying the wrong bits.
>
> Effectively no difference here (besides, in svn copy a file to another 
> directory is morally equivalent to a branch). If you use different 
> branches you still run the risk of failing to merge bits that aren't 
> obviously related.

There is a big difference if you think in terms of commits rather than in 
terms of files. If I merge a series of commits into another branch I can 
be sure that I got all the relevant changes and no more. Since, in my 
system, a single review request would be an atomic change e.g. the series 
of commits needed to add one testsuite, taking all the commits for a 
specific review and merging them onto the approved branch would give you a 
good assurance that you got the bits that had been reviewed but no more or 
less.

The fact that svn thinks of branches and directories as the same thing is 
a flaw in svn.

Received on Tuesday, 10 May 2011 21:16:02 UTC