Re: CfC: WebApps testing process; deadline April 20

On Wed, Apr 13, 2011 at 10:02 AM, Arthur Barstow <art.barstow@nokia.com> wrote:
> I have updated WebApps' testing process documents to reflect comments
> submitted to the initial draft process [1]. As such, this is a Call for
> Consensus to agree to the testing process as described in:
>
> http://www.w3.org/2008/webapps/wiki/Testing
> http://www.w3.org/2008/webapps/wiki/Submission
> http://www.w3.org/2008/webapps/wiki/Approval
> http://www.w3.org/2008/webapps/wiki/Harness

Sorry for the lateness of this review -- I was swamped with work and
didn't find the time to respond earlier.  I still have a few issues
with the proposed approval procedure.  The way it sounds is that tests
can either be non-approved, or approved.  Non-approved tests sound
like (I'm not totally clear on this) they're supposed to live in
per-contributor submission/ directories, without anyone else
necessarily having any say over them.  Approved tests, on the other
hand, only exist at LC or later, and can't be changed without Working
Group approval.  (Which means what?  I'm not sure.)

The problem I've seen with the submission/ vs. approved/ approach in
the HTMLWG is that there's no distinction between tests in submission/
that are useful and correct but just haven't been reviewed by anyone,
and tests that aren't complete or correct yet and aren't really
intended for serious use.  We should have a clear test suite that's
usable in practice well before LC, even if not all the tests are
reviewed yet.

So what I'd prefer is that the contents of approved/ be under the
control of the maintainer of the test suite, like the editor controls
the spec.  If people are submitting tests, the maintainer should be
allowed to approve them without a CfC, at least while the spec is
still a Working Draft.  That way we have a single repository from the
beginning that should include all useful tests, instead of having many
tests of varying quality scattered throughout submission/.  Once a
test is in approved/, it should be possible to file bug reports on it,
discuss it on the mailing list, etc.  It needs to be in a central
location and the responsibility of the working group, not the
submitter, so that the working group can ensure the test suite's
quality from the earliest possible point.

Then a CfC would be whether the WG is okay with the current contents
of approved/ or whether some of the tests should be un-approved.  A
CfC shouldn't be necessary to approve tests to begin with, any more
than it's necessary to make an edit to a spec.

Received on Sunday, 17 April 2011 23:07:26 UTC