Re: Getting vendors using W3C tests

On Tuesday 2011-05-31 12:52 +0200, James Graham wrote:
> * An internal problem with the way we store tests that makes
> updating external testsuites needlessly cumbersome. That is
> something I expect we will have to fix ourselves.
> 
> * Hardcoded paths to specific servers. External servers are
> poisonous to test reliability so we have to host all resources
> ourselves (it seems that Mozilla have come to the same conclusion
> [1]).
> 
> * Getting the results out of the test into our systems. In theory
> this is solved for testharness.js tests by including a second
> <script>; testharnessreport.js that is blank on the W3C site and
> which vendors can use to convert the testharness.js results to
> whatever is needed result-collection system they use. However not
> all tests have consistently included this extra script.
> 
> Out of the three problems we have the first we have to solve
> ourselves, the third can be solved by tightening review to ensure
> people include both scripts. The second problem needs a bit of
> thought. Possibly the testsuite could have a build step and servers
> could be variables in the original files. That has some
> disadvantages though since there is non-negligible overhead to
> making everyone build the testsuite before they can run it.

The second and third problem both seem amenable to automated
solutions.

For the second, what Mozilla has done is use a PAC (proxy
autoconfig) file during many of our tests.  It maps hosts at the
network layer all to one server.  The mistake we made there was not
redirecting all other hosts to someplace that wouldn't work.  A
similar solution seems like it could be useful here, though it's a
bit more work to do it cross-browser.

For the third, there should be a review script that complains when
tests don't include the necessary script, and it should be run
regularly and problems reported someplace that's likely to lead to
their being fixed.  (If we accumulate a collection of such scripts,
it would be good if it were easy to run them before committing a
test.)

-David

-- 
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

Received on Tuesday, 31 May 2011 11:31:57 UTC