Testing and Test Driver

Hi, folks-

We've talked about this a bit, but I'd like to reframe the whole testing 
issue.

tl;dr:
* we will need a lot of tests to properly test the Web Audio API
* we need tests early to ensure interoperability
* we need at least one individual to step up as "test driver"
* the entire WG should learn how to write tests, so we can effectively 
review them, and so we can all contribute



Long and Winding Road:

In order to get interoperable implementations, it's not enough to have a 
well-defined spec... we also need a well-developed test suite, which 
includes tests for each normative assertion in the spec.  This will 
probably amount to hundreds or even thousands of tests for a complex 
spec like the Web Audio API.

Traditionally, W3C Working Groups would develop tests during the 
Candidate Recommendation ("Implementation") phase, once the spec is 
stable, but experience has shown the flaws in this methodology:

1) When tests are developed only after the spec has stabilized, early 
implementers may have different interpretations of the spec, or may have 
unfound bugs, and it's much harder for content creators to write content 
that works across different browsers;

1a) When content is to different browsers with different or buggy 
behavior, it becomes harder to justify changing the implementation later 
to the correct behavior;

2) If the spec needs to change because of ambiguous wording that is only 
found during test-writing, it has to return to Last Call at least one 
more time, adding time and effort to completion;

3) The energy and participation of a Working Group during CR is usually 
low, because of the effort needed to get through Last Call, and because 
implementers are busy refining their implementations, and it takes much 
longer to develop a proper test suite. This is not only because test 
creation slows down, but because the even-less-pleasant task of test 
review (i.e. making sure the test tests the right thing, and only the 
right thing) slows down.

For these reasons, among others, it's much better to test early, as the 
spec is still in development.

I know that Google already has several tests that we can use, but we 
will need many more.


To oversee the creation and maintenance of the test suite, we need a 
Test Driver (aka Test Czar, Test Lead, Test Manager, or Test Editor); 
this person creates tests, trains others in test writing, gets other 
people to write tests, collects tests together, reviews tests, makes 
sure that there are enough tests for each testable assertion in the 
spec, helps review the spec to ensure that it's written in a way that 
facilitates testing, helps create and maintain the implementation report 
(the list of tests results for each User Agent), and so on.

This is a fairly new role in W3C WGs, but it's a critical one. Right 
now, we don't have one.  For more complex specs, like the Web Audio API, 
the Test Driver should be a different person than the spec editor.

So, if anyone on the WG is willing to step up to do this, please let us 
know.


Finally, in order to be a productive and effective group, we should all 
learn how to write tests for the Web Audio API, even if each of us makes 
only a few tests, or only simple ones. This will impress upon each of us 
the importance of testing, and more critically, will enable each of us 
to review tests.

I propose that we have a initial training session with Philippe Le 
Hégaret, the W3C Interaction Domain Lead and head of testing, in how 
best to write tests for this spec, and how to integrate those tests into 
W3C's Test Harness; we can also have follow-ups periodically to help us 
all along, so we can get comfortable with testing and get us in the 
habit. Each of us should come to the table with basic Javascript skills 
beforehand, but I expect that everyone in the Audio WG already knows JS.

As a side benefit, by the way, I've found that writing tests is 
sometimes a good way to settle disagreements on how something should be 
specified.


Thoughts?

Regards-
-Doug Schepers
W3C Developer Relations
Project Coordinator, SVG, WebApps, Touch Events, and Audio WGs

Received on Wednesday, 18 July 2012 23:00:26 UTC