- From: David Carlisle <davidc@nag.co.uk>
- Date: Tue, 30 Nov 2010 10:35:22 +0000
- To: James Graham <jgraham@opera.com>
- Cc: "'public-html-testsuite@w3.org'" <public-html-testsuite@w3.org>
On 30/11/2010 09:45, James Graham wrote: > I am looking for some feedback on the test harness script > testharness.js (note that this would better have been called a I've only written some very simplistic tests using this harness, but I have to say it worked as expected and the syntactic overhead required to wrap each test/assertion was no effort at all compared to the effort in thinking of what to test. The main things that I was confused about are perhaps due to lack of documentation (or not, I can't tell, hence the confusion) so some questions: When should this harness be used? (Many of the currently approved tests don't appear to use it.) How are results collected? The test produces a nice pass/fail test in display in the browser but is there is a mechanism for collecting the results from different test files and across different test runs, I assume so, but didn't understand how that was supposed to work. The harness (and the test suite generally) seems based on a binary pass/fail state for tests. The MathML and Xquery test suites had a standard reporting form (xml, but json would have done:-) and possible states for each test of pass/fail/partial-pass/broken/untested this allows combined results display such as http://www.w3.org/Math/testsuite/results/tests.html for mathml or the similar page for xquery http://dev.w3.org/2006/xquery-test-suite/PublicPagesStagingArea/XQTSReportSimple.html In both cases these tables can be easily combined with additional columns as each implementation submits a new result file. It wasn't clear to me how reporting of results was intended to work in this case. David ________________________________________________________________________ The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom. This e-mail has been scanned for all viruses by Star. The service is powered by MessageLabs. ________________________________________________________________________
Received on Tuesday, 30 November 2010 10:35:52 UTC