Re: RfC: WebApps Testing Process

On Thu, Mar 31, 2011 at 8:04 AM, Arthur Barstow <art.barstow@nokia.com> wrote:
> 3. http://www.w3.org/2008/webapps/wiki/Approval - how to start a test case
> review, approval process, how to update an approved test case

It looks like every submitted test suite must undergo a CfC, and so
must every update.  I'll first say that this is much better than the
way the HTMLWG currently works, which requires explicit endorsement by
a reviewer before approval but provides no way to obtain such review
other than hoping someone will be willing to give it.

But it's also much more cumbersome than the process for changing the
actual specifications.  In writing specs, instead, we designate one or
more people as editors for each spec, and let them make changes at
will.  Others who have suggestions can use the bug-tracking system,
and contentious issues must be addressed during Last Call before a
specification can become a Candidate Recommendation.  This procedure
is well-optimized for the fact that realistically, the large majority
of changes are uncontroversial, so it makes the most sense to assume
that changes are uncontroversial until someone actually objects,
without any formal approval requirements.

I propose that the Working Group follow this model for tests too.  For
each specification, we should designate certain people as maintainers
for that specification's tests.  They should be allowed to update the
tests and add new tests freely, and should be responsible for
responding to bug reports.  At Last Call, we should explicitly ask for
feedback for the test suite as well as the specification, and treat
feedback on tests similarly to feedback on the spec.

I suggest that for starters, the editors of a spec should also be
maintainers of its tests.  Additional test maintainers for each spec
can be added by a WG CfC.  This recognizes the fact that the editors
of a spec are presumably competent to judge tests, but that many
editors won't want to also review tests and many test reviewers won't
want to edit specs, so there's value in keeping the groups separate.

If the approval model currently on the wiki page is adopted, I predict
that it will become extremely cumbersome to maintain large test suites
for any specification that's not already very stable.  Any significant
specification change will require a CfC to change all affected test
cases, which will be a lot of spam to sift through if we have lots of
tests and spec changes.  This model also prevents editors who want to
write their own tests from writing the tests in conjunction with the
specification, because of the higher approval bar for tests than for
spec edits.  I don't think it makes sense at all.

Received on Friday, 1 April 2011 16:16:35 UTC