Re: RfC: WebApps Testing Process

Thanks for the comments Aryeh.

So rather than having a contribution reviewed and approved when the 
contributor wants to start a CfC to Review+Approved (R&A) their 
contribution, the idea is to allow any number of edits and contributions 
to a test suite until such time the group thinks the test suite is 
complete (e.g. after a CR is published) and then start a formal R&A? I 
like that idea.

I'm not sure we need to explicitly designate test suite maintainers. 
Ideally, someone other than the spec Editor(s) should create the test 
cases (e.g. to avoid the "fox guarding the chicken coup"). It seems like 
we should grant everyone CRUD rights to the test suites and if that 
becomes problematic we can address those issues later.

I'd like to see what others have to say about the original proposal and 
Aryeh's comments.

-Art Barstow

On Apr/1/2011 12:15 PM, ext Aryeh Gregor wrote:
> On Thu, Mar 31, 2011 at 8:04 AM, Arthur Barstow<art.barstow@nokia.com>  wrote:
>> 3. http://www.w3.org/2008/webapps/wiki/Approval - how to start a test case
>> review, approval process, how to update an approved test case
> It looks like every submitted test suite must undergo a CfC, and so
> must every update.  I'll first say that this is much better than the
> way the HTMLWG currently works, which requires explicit endorsement by
> a reviewer before approval but provides no way to obtain such review
> other than hoping someone will be willing to give it.
>
> But it's also much more cumbersome than the process for changing the
> actual specifications.  In writing specs, instead, we designate one or
> more people as editors for each spec, and let them make changes at
> will.  Others who have suggestions can use the bug-tracking system,
> and contentious issues must be addressed during Last Call before a
> specification can become a Candidate Recommendation.  This procedure
> is well-optimized for the fact that realistically, the large majority
> of changes are uncontroversial, so it makes the most sense to assume
> that changes are uncontroversial until someone actually objects,
> without any formal approval requirements.
>
> I propose that the Working Group follow this model for tests too.  For
> each specification, we should designate certain people as maintainers
> for that specification's tests.  They should be allowed to update the
> tests and add new tests freely, and should be responsible for
> responding to bug reports.  At Last Call, we should explicitly ask for
> feedback for the test suite as well as the specification, and treat
> feedback on tests similarly to feedback on the spec.
>
> I suggest that for starters, the editors of a spec should also be
> maintainers of its tests.  Additional test maintainers for each spec
> can be added by a WG CfC.  This recognizes the fact that the editors
> of a spec are presumably competent to judge tests, but that many
> editors won't want to also review tests and many test reviewers won't
> want to edit specs, so there's value in keeping the groups separate.
>
> If the approval model currently on the wiki page is adopted, I predict
> that it will become extremely cumbersome to maintain large test suites
> for any specification that's not already very stable.  Any significant
> specification change will require a CfC to change all affected test
> cases, which will be a lot of spam to sift through if we have lots of
> tests and spec changes.  This model also prevents editors who want to
> write their own tests from writing the tests in conjunction with the
> specification, because of the higher approval bar for tests than for
> spec edits.  I don't think it makes sense at all.

Received on Friday, 1 April 2011 17:27:25 UTC