Re: Test review procedure

On Thursday 2011-03-17 01:32 +0000, Kris Krueger wrote:
> The btoa and atob tests are not part of the spec so they can't be approved.
> 
> http://www.w3.org/Bugs/Public/show_bug.cgi?id=12029
> 
> Ian 'Hixie' Hickson 2011-02-10 19:44:26 UTC 
> Oops, my bad. I didn't mean to include window.atob() in the W3C copy. Will take
> care of that.
> 
> Now for the reflection tests - I'd suggest the following to get consensus.
> 
> First refactor the tests so that it tests a few attributes to start rather than the current huge list.
> The way the tests are currently factored it requires every test to be reviewed before any can get approved.

You seem to be ignoring the bigger point, which is that we shouldn't
have this review process in the first place.

I thought there was consensus at the testing meeting at the plenary
that we shouldn't have a review process like this.  Instead, we
should accept tests eagerly and allow mistakes to be challenged
easily.

Why do we still have a review process?  And what does it take to get
it changed?

-David


> -----Original Message-----
> From: public-html-testsuite-request@w3.org [mailto:public-html-testsuite-request@w3.org] On Behalf Of Aryeh Gregor
> Sent: Wednesday, March 16, 2011 3:37 PM
> To: public-html-testsuite@w3.org
> Subject: Test review procedure
> 
> Current procedure appears to be that submitted tests must be reviewed by an independent party before they're approved.  However, there is apparently no procedure to ensure that they ever get reviewed, which means it's possible for submitted tests to get stuck in limbo.  That's particularly likely if the tests are very extensive, because then no one will want to review them fully.  This is not a good thing if the goal is to produce a comprehensive test suite.  Small tests might be easy to approve, but tests with thousands of separate assertions will not be easy to review.
> 
> We don't require this high a standard of review for the specification itself, so it's not reasonable to require it for the tests.  A specification doesn't need anyone to formally sign off on it to progress along the Recommendation track -- if no one objects to a particular part of the specification, it's assumed to be okay.
> Likewise, if tests are posted for review and no one objects within a fixed timeframe, let's say thirty days, they should be approved without review.  If anyone subsequently finds errors and the tests aren't fixed quickly, they can be un-approved at that point.  This is no different from how errors can be introduced into the specification itself.
> 
> Specifically, I've submitted tests for two features so far: base64 functions on February 9 <http://lists.w3.org/Archives/Public/public-html-testsuite/2011Feb/0013.html>,
> and reflection on February 16
> <http://lists.w3.org/Archives/Public/public-html-testsuite/2011Feb/0029.html>.
>  I received no responses to my request for reviewing the base64 tests.
>  I received two responses to my reflection tests, both of which I responded to in detail within a day, and have gotten no further responses.  It's been a month now for the reflection tests, and more for the base64 tests.  Under the current system, I have no reason to be sure that these tests will ever be approved, particularly not the reflection tests.
> 
> 

-- 
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

Received on Thursday, 17 March 2011 05:52:23 UTC