Getting vendors using W3C tests

So, the most important feature of a testsuite is that it is able to 
catch implementation bugs in a timely fashion. What this means in 
practice is that the tests must be run as part of the regular regression 
checking setup that the implementer uses for its development builds. So 
far W3C testsuites have not been a success in this regard. For example 
the CSS2.1 testsuite was, as far as I can tell, not being regularly run 
by (m)any implementers right up until the final push for PR. Indeed, 
even now I think that it is not being run by some implementers due to 
the over-reliance on tests that require visual interpretation of the 
results. Similarly I am not aware that anyone is running the HTML5 
testsuite in their nightly builds (I hope I am wrong).

This has to change.

So, the question is "what actions can we take to make vendors run the 
W3C testsuites as part of their regular regression builds". Interesting 
things are those that work cross-vendor so suggestions like "use X 
testing framework that we (and no one else) already use" probably aren't 
that helpful, unless use of that framework allows a step change in the 
kinds of things that can be automated (visual tests to reftests for 
example).

At Opera, the problems that we currently have are (off the top of my 
head, someone else might add more):

* An internal problem with the way we store tests that makes updating 
external testsuites needlessly cumbersome. That is something I expect we 
will have to fix ourselves.

* Hardcoded paths to specific servers. External servers are poisonous to 
test reliability so we have to host all resources ourselves (it seems 
that Mozilla have come to the same conclusion [1]).

* Getting the results out of the test into our systems. In theory this 
is solved for testharness.js tests by including a second <script>; 
testharnessreport.js that is blank on the W3C site and which vendors can 
use to convert the testharness.js results to whatever is needed 
result-collection system they use. However not all tests have 
consistently included this extra script.

Out of the three problems we have the first we have to solve ourselves, 
the third can be solved by tightening review to ensure people include 
both scripts. The second problem needs a bit of thought. Possibly the 
testsuite could have a build step and servers could be variables in the 
original files. That has some disadvantages though since there is 
non-negligible overhead to making everyone build the testsuite before 
they can run it.

[1] 
https://developer.mozilla.org/en/QA/Avoiding_intermittent_oranges#Tests_that_rely_on_external_sites

Received on Tuesday, 31 May 2011 10:54:05 UTC