Test review procedure

Current procedure appears to be that submitted tests must be reviewed
by an independent party before they're approved.  However, there is
apparently no procedure to ensure that they ever get reviewed, which
means it's possible for submitted tests to get stuck in limbo.  That's
particularly likely if the tests are very extensive, because then no
one will want to review them fully.  This is not a good thing if the
goal is to produce a comprehensive test suite.  Small tests might be
easy to approve, but tests with thousands of separate assertions will
not be easy to review.

We don't require this high a standard of review for the specification
itself, so it's not reasonable to require it for the tests.  A
specification doesn't need anyone to formally sign off on it to
progress along the Recommendation track -- if no one objects to a
particular part of the specification, it's assumed to be okay.
Likewise, if tests are posted for review and no one objects within a
fixed timeframe, let's say thirty days, they should be approved
without review.  If anyone subsequently finds errors and the tests
aren't fixed quickly, they can be un-approved at that point.  This is
no different from how errors can be introduced into the specification
itself.

Specifically, I've submitted tests for two features so far: base64
functions on February 9
<http://lists.w3.org/Archives/Public/public-html-testsuite/2011Feb/0013.html>,
and reflection on February 16
<http://lists.w3.org/Archives/Public/public-html-testsuite/2011Feb/0029.html>.
 I received no responses to my request for reviewing the base64 tests.
 I received two responses to my reflection tests, both of which I
responded to in detail within a day, and have gotten no further
responses.  It's been a month now for the reflection tests, and more
for the base64 tests.  Under the current system, I have no reason to
be sure that these tests will ever be approved, particularly not the
reflection tests.

Received on Wednesday, 16 March 2011 22:38:21 UTC