Re: Request for Feedback On Test Harness

On 30/11/2010 09:45, James Graham wrote:
> I am looking for some feedback on the test harness script
> testharness.js (note that this would better have been called a

I've only written some very simplistic tests using this harness, but I 
have to say it worked as expected and the syntactic overhead required to 
wrap each test/assertion was no effort at all compared to the effort in 
thinking of what to test.

The main things that I was confused about are perhaps due to lack of 
documentation (or not, I can't tell, hence the confusion) so some questions:

When should this harness be used? (Many of the currently approved tests 
don't appear to use it.)

How are results collected? The test produces a nice pass/fail test in 
display in the browser but is there is a mechanism for collecting the 
results from different test files and across different test runs, I 
assume so, but didn't understand how that was supposed to work.

The harness (and the test suite generally) seems based on a binary 
pass/fail state for tests.  The MathML and Xquery test suites had a 
standard reporting form (xml, but json would have done:-) and possible 
states for each test of pass/fail/partial-pass/broken/untested this 
allows combined results display such as

http://www.w3.org/Math/testsuite/results/tests.html

for mathml

or the similar page for xquery

http://dev.w3.org/2006/xquery-test-suite/PublicPagesStagingArea/XQTSReportSimple.html

In both cases these tables can be easily combined with additional 
columns as each implementation submits a new result file. It wasn't 
clear to me how reporting of results was intended to work in this case.



David








________________________________________________________________________
The Numerical Algorithms Group Ltd is a company registered in England
and Wales with company number 1249803. The registered office is:
Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.

This e-mail has been scanned for all viruses by Star. The service is
powered by MessageLabs. 
________________________________________________________________________

Received on Tuesday, 30 November 2010 10:35:52 UTC