Fwd: Testing and the WebApps working group

-------- Original Message --------
Subject: Testing and the WebApps working group
Date: Fri, 25 Feb 2011 18:00:34 +0000
From: Adrian Bateman <adrianba@microsoft.com>

Gentlemen, you will recall that at TPAC we discussed testing for WebApps 
specs and our desire
for a more formal process that provides an element of consistency to the 
numerous specs created
in the working group. Microsoft is keen to contribute to the testing 
process within a more
structured framework. We have uploaded tests for DOM L3 Events but there 
is no process in place
for getting these reviewed and approved. We think the work happening in 
the testing task force
of the HTML working group is a good model.

At TPAC I volunteered to help the chairs and the Team to come up with a 
proposal to present to
the working group for discussion. I don't know if there has been any 
progress on this so I've
started an outline (below) to begin the conversation.

It would perhaps be helpful to have a testing facilitator in the same 
way that Kris Krueger helps
in the HTML WG. Since Microsoft is performing this role for HTML, we 
don't think that it would be
wise to volunteer for this in WebApps too but we would be happy to help 
in the interim to get
things started.

Please let me know how you think we can make progress here and how I can 
help.

Thanks,

Adrian.


SUBMIT TESTS

1) For working group members:
    Submits tests to Mercurial 'webapps' repository in the 
tests/submission/<vendor> folder.

2) For submitters who are not working group members:
    a) Fill out the W3C license grant form
    b) Make the tests available at a public URL and send e-mail to 
public-webapps describing
       the tests being submitted and giving permission for their use as 
described in the grant
       of license.

Tests should follow the HTML5 test harness [1] (ideally using the 
JavaScript harness [2]).

REVIEW AND APPROVAL OF TESTS

Members of the WG review the tests and reply to the list with any 
disagreements/objections
to the tests.

Periodically (every 2 months?) tests are moved to the approval folder as 
approved tests:

1) All tests that have had no objections and were submitted at least 2 
weeks before the
    approval date become approved (added to the tests/approved folder). 
The chairs may
    vary this if a large number of tests were submitted at the same time 
and they judge
    it unreasonable for them all to have been reviewed.

2) If there is a discrepancy between the tests and the spec, discussion 
should lead to
    consensus about whether the spec or the test needs to change. Once 
this correction is
    made, these tests are approved at the next approval date.

3) If new information comes to light suggesting that a test is invalid a 
bug is filed
    against the test. This is investigated and, if confirmed, the test 
is suspended from
    the approved test suite until it is corrected.

The goal of this process is that tests are not held up waiting for 
review but that everyone
has an opportunity to review and object to tests that they disagree with.

[1] http://www.w3.org/html/wg/wiki/Testing#W3C_Official_HTML5_Test_Cases
[2] http://test.w3.org/html/tests/approved/getElementsByClassName/001.htm

-- 
Adrian Bateman
Program Manager - Internet Explorer - Microsoft Corporation
Phone: +1 (425) 538 5111
Email: mailto:adrianba@microsoft.com

Received on Monday, 28 February 2011 02:43:57 UTC