Re: CfC: WebApps testing process; deadline April 20

Thanks for the feedback!

Yes, I agree early and thorough review is needed and my expectation 
was/is that those vested in a spec and its test suite would actively 
participate in the creation and review of tests, regardless of whether 
that function was documented or not. I will add some related text to the 
wiki in a day or two. I think the gist of the review and approve process 
is that a contributor or test suite maintainer may start a Request for 
Review (RfR) on public-webapps-testsuite (p-w-ts) for some set of test 
cases and if no issues are raised the maintainer will  copy the test 
cases to the approved directory. I can imagine some test suites having 
multiple RfRs. After a test suite is approved by the p-w-ts community 
and the WG considers its spec complete, I think it would then make sense 
to have a formal CfC among the entire WG to approve a test suite.

Re various "roles", I agree they should be clarified. Below is a rough 
draft and comments are welcome. If someone wants to lead WebApps' test 
coordination, please let me know.

Re clear test suite status, below is a rough draft for that section and 
comments on that are also welcome, especially pointers to the process 
and mechanisms others use to track test case status.

-ArtB

== Roles ==

* Contributor - someone that contributes one or more test cases

* Test Suite Maintainer - the person(s) responsible for maintaining a 
specification's test suite, including: updating test cases, maintaining 
the test suite's status, etc. Normally,  this role is assumed by the 
specification's Editor(s).

* Testing Coordinator - the person responsible for coordinating all of 
WebApps' testing efforts, including: assuring only approved tests are 
put in <code>.../approved/</code> directories, making sure the test 
suites' status are current, etc. By default, this role is taken by a WG 
Chair, a WebApps '''Team Contact''' or a member of the WG. (In practice, 
this role may be shared by more than one person.)

== Test Suite Status ==

To facilitate determining the status of a test suite, a Test Suite 
Maintainer is responsible for maintaining the test suite's status. Each 
test suite includes a '''Status''' file i.e. 
<code>.../tests/Status.html</code> and this file includes at least the 
following information:

# Test Suite Maintainer(s): <names ...>
# Approval status: normally, one of: a) no test cases are approved; b) 
test cases in the <code>.../approved/</code> directory are approved by 
members of WebApps' test group (public-webapps-testsuite); c) the WG has 
formally approved all of the test cases in the 
<code>.../approved/</code> directory.


On Apr/20/2011 7:10 PM, ext Adrian Bateman wrote:
> First, thanks to Art for pulling all this content together. We're looking
> forward to a more structured process for testing as various specifications
> in the WebApps increase in maturity.
>
> I have a couple of small comments related to the issues Aryeh raised.
> Apologies for the lateness of these comments; I spent time sharing this
> process with a number of teams at Microsoft before responding.
>
> 1. Early approval of tests
>
> We think that waiting for Last Call or Candidate Recommendation before
> approving tests loses some of the value of tests in driving ambiguity out
> of the specification. The CSS WG found many issues in CSS 2.1 as a
> consequence of tests and some of these issues we substantive enough that
> the spec went back to Working Draft status. Avoiding this by reviewing test
> cases earlier in the process will help to improve spec quality. I think
> of this in the following way: a bug filed against the spec requesting a
> change represents someone's view that the spec is wrong. On the other hand,
> an approved test with consensus of reviewers in the working group helps to
> identify more stable sections of the spec. It doesn't mean it can't change
> but it does mean the spec has had additional review on the assertions made
> in the test and that's useful.
>
> 2. Calls for Consensus
>
> When a contributor submits a set of tests, hopefully members of the
> public-webapps-testsuite list will review them. When the contributor believes
> they have addressed all of the feedback on the tests, we think they should
> be able to put the tests forward for a Call for Consensus on approving them.
> The WG may choose to issue such a call on a regular schedule for tests that
> have had sufficient time for review if the number of submissions makes doing
> this "on demand" too cumbersome. This call may just be made to the testsuite
> mailing list. A full WG consensus shouldn't be necessary until document
> transition.
>
> 3. Test suite maintainer
>
> It's not clear from the discussion below what the role of the maintainer is
> and whether there's one per spec or one for the group. This seems like largely
> an administrative role in ensuring that the appropriate tests are moved into
> the approval folder at the correct time and that the status of the test suite
> is accurately recorded. Perhaps members of the working group could volunteer
> to help with this or maybe the Team could help. Microsoft would be willing
> to contribute here too.
>
> Thanks,
>
> Adrian.
>
> On Tuesday, April 19, 2011 9:22 AM, Arthur Barstow wrote:
>> I agree the need for clear test suite status is implied and should be
>> explicit. I've added a new requirement for this to [1]. As to how this
>> requirement is addressed, perhaps we should adopt/re-use some existing
>> good practice; otherwise perhaps we can use a Status/Readme file in each
>> .../tests/ directory.
>>
>> Besides "WG Approval", you see the need for "maintainer approval".
>> Additionally, you think there will be scenarios where multiple
>> submissions make it difficult to know which tests have been reviewed
>> (e.g. by the maintainer). This could be addressed by the maintainer
>> creating a separate directory (e.g. "reviewed") and the maintainer would
>> use it to place copies of test cases (e.g. from submissions directories)
>> she reviewed (and possibly edited and approved). I don't think the
>> process as currently defined would preclude the maintainer from doing
>> something like this and if it becomes a common/good practice, we could
>> document it. In the case of a single contributor, it's probably not
>> something that would be needed (but wouldn't necessarily be harmful).
>>
>> -Art Barstow
>>
>> [1] http://www.w3.org/2008/webapps/wiki/Testing#Testing_Goals
>>
>> On Apr/17/2011 7:06 PM, ext Aryeh Gregor wrote:
>>> On Wed, Apr 13, 2011 at 10:02 AM, Arthur Barstow<art.barstow@nokia.com>
>> wrote:
>>>> I have updated WebApps' testing process documents to reflect comments
>>>> submitted to the initial draft process [1]. As such, this is a Call
>> for
>>>> Consensus to agree to the testing process as described in:
>>>>
>>>> http://www.w3.org/2008/webapps/wiki/Testing
>>>> http://www.w3.org/2008/webapps/wiki/Submission
>>>> http://www.w3.org/2008/webapps/wiki/Approval
>>>> http://www.w3.org/2008/webapps/wiki/Harness
>>> Sorry for the lateness of this review -- I was swamped with work and
>>> didn't find the time to respond earlier.  I still have a few issues
>>> with the proposed approval procedure.  The way it sounds is that tests
>>> can either be non-approved, or approved.  Non-approved tests sound
>>> like (I'm not totally clear on this) they're supposed to live in
>>> per-contributor submission/ directories, without anyone else
>>> necessarily having any say over them.  Approved tests, on the other
>>> hand, only exist at LC or later, and can't be changed without Working
>>> Group approval.  (Which means what?  I'm not sure.)
>>>
>>> The problem I've seen with the submission/ vs. approved/ approach in
>>> the HTMLWG is that there's no distinction between tests in submission/
>>> that are useful and correct but just haven't been reviewed by anyone,
>>> and tests that aren't complete or correct yet and aren't really
>>> intended for serious use.  We should have a clear test suite that's
>>> usable in practice well before LC, even if not all the tests are
>>> reviewed yet.
>>>
>>> So what I'd prefer is that the contents of approved/ be under the
>>> control of the maintainer of the test suite, like the editor controls
>>> the spec.  If people are submitting tests, the maintainer should be
>>> allowed to approve them without a CfC, at least while the spec is
>>> still a Working Draft.  That way we have a single repository from the
>>> beginning that should include all useful tests, instead of having many
>>> tests of varying quality scattered throughout submission/.  Once a
>>> test is in approved/, it should be possible to file bug reports on it,
>>> discuss it on the mailing list, etc.  It needs to be in a central
>>> location and the responsibility of the working group, not the
>>> submitter, so that the working group can ensure the test suite's
>>> quality from the earliest possible point.
>>>
>>> Then a CfC would be whether the WG is okay with the current contents
>>> of approved/ or whether some of the tests should be un-approved.  A
>>> CfC shouldn't be necessary to approve tests to begin with, any more
>>> than it's necessary to make an edit to a spec.

Received on Thursday, 21 April 2011 16:44:01 UTC