Re: Test review procedure

On Wed, 16 Mar 2011, L. David Baron wrote:

> On Thursday 2011-03-17 01:32 +0000, Kris Krueger wrote:
>> The btoa and atob tests are not part of the spec so they can't be approved.
>>
>> http://www.w3.org/Bugs/Public/show_bug.cgi?id=12029
>>
>> Ian 'Hixie' Hickson 2011-02-10 19:44:26 UTC
>> Oops, my bad. I didn't mean to include window.atob() in the W3C copy. Will take
>> care of that.
>>
>> Now for the reflection tests - I'd suggest the following to get consensus.
>>
>> First refactor the tests so that it tests a few attributes to start rather than the current huge list.
>> The way the tests are currently factored it requires every test to be reviewed before any can get approved.
>
> You seem to be ignoring the bigger point, which is that we shouldn't
> have this review process in the first place.

I guess I should also respond to the bigger point. Some observations:

a) The current review procedure is clearly not working as expected. Large 
contributions are typically being taken on an accept-then-review basis 
despite the general policy of review-then-accept. This happened with 
Philip's cnavas tests, for example.

b) A substantial reason for the above seems to be a lack of interest from 
browser vendors in performing review. So far most review has been 
performed, as far as I can tell, by individuals. Having a process that 
people aren't interested in contributing to seems problematic. Having said 
that, I wonder if people are actually using the HTMLWG tests as part of 
their regression test suites yet. Perhaps if they start to use the tests 
there will be more incentive to perform review. It is also possible that 
they will only look at the tests they fail and ignore the ones they pass.

c) In spite of the above, the review that has happened has uncovered a 
non-trivial number of bugs.

d) The CSS 2.1 testsuite has had a lot of churn just before PR because 
people suddenly started taking it seriously and reporting problems. This 
is not good for anyone and we should try to avoid the same thing 
happening.

e) Trying to address d) by having continually updated implementation 
reports has a number of bad side effect; it encourages the erroneous use 
of the testsuite as a metric of quality when it is a largely incomplete 
state. This in turn can increase the pressure on potential contributers to 
submit only tests that cast their favoured browser in a good light.

f) It occasionally happens that third parties want to take W3C tests and 
reuse them in some way. Often in these cases it is beneficial to everyone 
if the third party can be pointed to tests that are known to be reasonably 
bug free, and cover sections of the spec that are stable. Review doesn't 
help with the latter but it can help with the former.

So my conclusion is that the process as-is doesn't work optimally, but I 
am not clear what the best way forward is.

Received on Friday, 18 March 2011 08:25:35 UTC