RE: CfC: WebApps testing process; deadline April 20

First, thanks to Art for pulling all this content together. We're looking
forward to a more structured process for testing as various specifications
in the WebApps increase in maturity.

I have a couple of small comments related to the issues Aryeh raised.
Apologies for the lateness of these comments; I spent time sharing this
process with a number of teams at Microsoft before responding.

1. Early approval of tests

We think that waiting for Last Call or Candidate Recommendation before
approving tests loses some of the value of tests in driving ambiguity out
of the specification. The CSS WG found many issues in CSS 2.1 as a
consequence of tests and some of these issues we substantive enough that
the spec went back to Working Draft status. Avoiding this by reviewing test
cases earlier in the process will help to improve spec quality. I think
of this in the following way: a bug filed against the spec requesting a
change represents someone's view that the spec is wrong. On the other hand,
an approved test with consensus of reviewers in the working group helps to
identify more stable sections of the spec. It doesn't mean it can't change
but it does mean the spec has had additional review on the assertions made
in the test and that's useful.

2. Calls for Consensus

When a contributor submits a set of tests, hopefully members of the
public-webapps-testsuite list will review them. When the contributor believes
they have addressed all of the feedback on the tests, we think they should
be able to put the tests forward for a Call for Consensus on approving them.
The WG may choose to issue such a call on a regular schedule for tests that
have had sufficient time for review if the number of submissions makes doing
this "on demand" too cumbersome. This call may just be made to the testsuite
mailing list. A full WG consensus shouldn't be necessary until document
transition.

3. Test suite maintainer

It's not clear from the discussion below what the role of the maintainer is
and whether there's one per spec or one for the group. This seems like largely
an administrative role in ensuring that the appropriate tests are moved into
the approval folder at the correct time and that the status of the test suite
is accurately recorded. Perhaps members of the working group could volunteer
to help with this or maybe the Team could help. Microsoft would be willing
to contribute here too.

Thanks,

Adrian.

On Tuesday, April 19, 2011 9:22 AM, Arthur Barstow wrote:
> I agree the need for clear test suite status is implied and should be
> explicit. I've added a new requirement for this to [1]. As to how this
> requirement is addressed, perhaps we should adopt/re-use some existing
> good practice; otherwise perhaps we can use a Status/Readme file in each
> .../tests/ directory.
> 
> Besides "WG Approval", you see the need for "maintainer approval".
> Additionally, you think there will be scenarios where multiple
> submissions make it difficult to know which tests have been reviewed
> (e.g. by the maintainer). This could be addressed by the maintainer
> creating a separate directory (e.g. "reviewed") and the maintainer would
> use it to place copies of test cases (e.g. from submissions directories)
> she reviewed (and possibly edited and approved). I don't think the
> process as currently defined would preclude the maintainer from doing
> something like this and if it becomes a common/good practice, we could
> document it. In the case of a single contributor, it's probably not
> something that would be needed (but wouldn't necessarily be harmful).
> 
> -Art Barstow
> 
> [1] http://www.w3.org/2008/webapps/wiki/Testing#Testing_Goals
> 
> On Apr/17/2011 7:06 PM, ext Aryeh Gregor wrote:
> > On Wed, Apr 13, 2011 at 10:02 AM, Arthur Barstow<art.barstow@nokia.com>
> wrote:
> >> I have updated WebApps' testing process documents to reflect comments
> >> submitted to the initial draft process [1]. As such, this is a Call
> for
> >> Consensus to agree to the testing process as described in:
> >>
> >> http://www.w3.org/2008/webapps/wiki/Testing
> >> http://www.w3.org/2008/webapps/wiki/Submission
> >> http://www.w3.org/2008/webapps/wiki/Approval
> >> http://www.w3.org/2008/webapps/wiki/Harness
> > Sorry for the lateness of this review -- I was swamped with work and
> > didn't find the time to respond earlier.  I still have a few issues
> > with the proposed approval procedure.  The way it sounds is that tests
> > can either be non-approved, or approved.  Non-approved tests sound
> > like (I'm not totally clear on this) they're supposed to live in
> > per-contributor submission/ directories, without anyone else
> > necessarily having any say over them.  Approved tests, on the other
> > hand, only exist at LC or later, and can't be changed without Working
> > Group approval.  (Which means what?  I'm not sure.)
> >
> > The problem I've seen with the submission/ vs. approved/ approach in
> > the HTMLWG is that there's no distinction between tests in submission/
> > that are useful and correct but just haven't been reviewed by anyone,
> > and tests that aren't complete or correct yet and aren't really
> > intended for serious use.  We should have a clear test suite that's
> > usable in practice well before LC, even if not all the tests are
> > reviewed yet.
> >
> > So what I'd prefer is that the contents of approved/ be under the
> > control of the maintainer of the test suite, like the editor controls
> > the spec.  If people are submitting tests, the maintainer should be
> > allowed to approve them without a CfC, at least while the spec is
> > still a Working Draft.  That way we have a single repository from the
> > beginning that should include all useful tests, instead of having many
> > tests of varying quality scattered throughout submission/.  Once a
> > test is in approved/, it should be possible to file bug reports on it,
> > discuss it on the mailing list, etc.  It needs to be in a central
> > location and the responsibility of the working group, not the
> > submitter, so that the working group can ensure the test suite's
> > quality from the earliest possible point.
> >
> > Then a CfC would be whether the WG is okay with the current contents
> > of approved/ or whether some of the tests should be un-approved.  A
> > CfC shouldn't be necessary to approve tests to begin with, any more
> > than it's necessary to make an edit to a spec.
>

Received on Wednesday, 20 April 2011 23:12:04 UTC