Re: Request for Feedback On Test Harness

On 30/11/2010 09:45, James Graham wrote:
> I am looking for some feedback on the test harness script
> testharness.js (note that this would better have been called a

I've only written some very simplistic tests using this harness, but I 
have to say it worked as expected and the syntactic overhead required to 
wrap each test/assertion was no effort at all compared to the effort in 
thinking of what to test.

The main things that I was confused about are perhaps due to lack of 
documentation (or not, I can't tell, hence the confusion) so some questions:

When should this harness be used? (Many of the currently approved tests 
don't appear to use it.)

How are results collected? The test produces a nice pass/fail test in 
display in the browser but is there is a mechanism for collecting the 
results from different test files and across different test runs, I 
assume so, but didn't understand how that was supposed to work.

The harness (and the test suite generally) seems based on a binary 
pass/fail state for tests.  The MathML and Xquery test suites had a 
standard reporting form (xml, but json would have done:-) and possible 
states for each test of pass/fail/partial-pass/broken/untested this 
allows combined results display such as

for mathml

or the similar page for xquery

In both cases these tables can be easily combined with additional 
columns as each implementation submits a new result file. It wasn't 
clear to me how reporting of results was intended to work in this case.


The Numerical Algorithms Group Ltd is a company registered in England
and Wales with company number 1249803. The registered office is:
Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.

This e-mail has been scanned for all viruses by Star. The service is
powered by MessageLabs. 

Received on Tuesday, 30 November 2010 10:35:52 UTC