Re: Approval process for DOM4 tests

On 1/17/12 12:00 PM, ext Aryeh Gregor wrote:
> Currently there are a number of DOM4 tests (primarily Range-related)
> that are undergoing active revision.  The tests are fairly
> complicated, and bugs tend to crop up, e.g.:
>
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=15583
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=15584
>
> Also, I'm expanding and revising some of the tests on an ongoing
> basis.  Previously we had a Request for Review of a bunch of submitted
> tests, which had zero responses:
>
> http://lists.w3.org/Archives/Public/public-webapps-testsuite/2011Nov/0008.html
>
> I suspect that zero responses will be typical for such RfRs.  Bugs
> will be found as implementers actually try to pass all the tests, as
> Opera has been doing recently.  On the other hand, having to wait for
> an RfR before approving test changes is cumbersome.
>
>
> Thus I suggest that for DOM4, tests can be moved to the approved/
> directory by any DOM4 editor at their discretion, the same as the spec
> itself can be changed.  The tests can be reviewed and approved by the
> Working Group at a later date, at the same time as the spec itself.
> When the spec becomes finalized at the PR stage, the tests can be
> finalized too, or at least stabilized (with changes requiring WG
> approval).  But for now they should just be maintained by the editors,
> like the spec.
>
> Does anyone object to this?  If not, I'll update Status.html.

So I think there are a few issues here ...

The first one is that it is too painful for a Test Facilitator or spec 
Editor to do a RfR for fixes/patches for tests already approved by 
WebApps' Testing Group. As such, the RfR requirement for patches should 
be relaxed, especially for the Test Facilitator(s) and Editor(s).

I think the proposal for Test Facilitators to be able to submit patches 
to approved tests without an explicit RfR makes sense. After all, the 
Approval process [Approval] does include a WG-wide CfC after the testing 
group considers the test suite complete. However, to provide some 
transparency here, I think the list should be notified after patches by 
Test Facilitators or Editors are submitted to approved tests. Would that 
be sufficient to address this issue?

The second issue is whether a RfR for new tests should be required if 
the submitter is a Test Facilitator or spec Editor.

I can see how this is effectively "make work" if no one other than the 
Facilitator or Editor is going to actually review the tests. OTOH, I 
think there is some value in having a transparent call for review (e.g. 
we never know who may have some comments). We could relax this 
requirement a bit if we allowed a Facilitator/Editor to copy tests to 
the approved directory and then require them to send a RfR that 
essentially says "I just copied <insert test list> to the approved 
directory. If you have any comments, submit them by T+7days". Would 
something like this be sufficient to address this issue?

The other issue is about how we should interpret silence on. This is an 
interesting question and I open to suggestions. We have two reviews to 
consider: a) a RfR by WebApps' test group for some set of tests; and b) 
a formal CfC by the entire WG for a test suite. For a), I tend to think 
we should continue to interpret silence on a RfR as a NOOP. However, for 
the later, perhaps we should consider something more rigorous. For 
instance, since we require 2 or more implementations of the spec to 
advance it to PR, perhaps we should also require at least 2 reviews for 
each test before the spec advances to PR. WDYT?

-AB

[Approval] http://www.w3.org/2008/webapps/wiki/Approval

Received on Thursday, 19 January 2012 17:26:34 UTC