RE: Test review procedure

The btoa and atob tests are not part of the spec so they can't be approved.

http://www.w3.org/Bugs/Public/show_bug.cgi?id=12029


Ian 'Hixie' Hickson 2011-02-10 19:44:26 UTC 
Oops, my bad. I didn't mean to include window.atob() in the W3C copy. Will take
care of that.

Now for the reflection tests - I'd suggest the following to get consensus.

First refactor the tests so that it tests a few attributes to start rather than the current huge list.
The way the tests are currently factored it requires every test to be reviewed before any can get approved.

-Kris

-----Original Message-----
From: public-html-testsuite-request@w3.org [mailto:public-html-testsuite-request@w3.org] On Behalf Of Aryeh Gregor
Sent: Wednesday, March 16, 2011 3:37 PM
To: public-html-testsuite@w3.org
Subject: Test review procedure

Current procedure appears to be that submitted tests must be reviewed by an independent party before they're approved.  However, there is apparently no procedure to ensure that they ever get reviewed, which means it's possible for submitted tests to get stuck in limbo.  That's particularly likely if the tests are very extensive, because then no one will want to review them fully.  This is not a good thing if the goal is to produce a comprehensive test suite.  Small tests might be easy to approve, but tests with thousands of separate assertions will not be easy to review.

We don't require this high a standard of review for the specification itself, so it's not reasonable to require it for the tests.  A specification doesn't need anyone to formally sign off on it to progress along the Recommendation track -- if no one objects to a particular part of the specification, it's assumed to be okay.
Likewise, if tests are posted for review and no one objects within a fixed timeframe, let's say thirty days, they should be approved without review.  If anyone subsequently finds errors and the tests aren't fixed quickly, they can be un-approved at that point.  This is no different from how errors can be introduced into the specification itself.

Specifically, I've submitted tests for two features so far: base64 functions on February 9 <http://lists.w3.org/Archives/Public/public-html-testsuite/2011Feb/0013.html>,
and reflection on February 16
<http://lists.w3.org/Archives/Public/public-html-testsuite/2011Feb/0029.html>.
 I received no responses to my request for reviewing the base64 tests.
 I received two responses to my reflection tests, both of which I responded to in detail within a day, and have gotten no further responses.  It's been a month now for the reflection tests, and more for the base64 tests.  Under the current system, I have no reason to be sure that these tests will ever be approved, particularly not the reflection tests.

Received on Thursday, 17 March 2011 01:32:46 UTC