Re: Request for Feedback On Test Harness

On 30/11/2010 10:53, James Graham wrote:
> The difference between "fail" and "broken" seems non-obvious to me; one
> would not expect test files that are inherently broken to be long-lived.

No but it is related to the way the reporting happens. The model for 
mathml and xquery test suites is that each implementation does it's own 
run and just reports the results as a file.

This allows merging of results from different operating systems, and 
from internal unreleased development builds or that have been done at 
different times.

So if you want to report the result of a couple of thousand tests but 
think that half a dozen of the tests were wrong, being able to put 
"broken" for the state on those lines is very convenient way of flagging 
those tests to be looked at. Agreed, in the end, the tests got fixed (or 
classified as OK) and so the final test results should say pass or fail.

> And "some-passed" seems like "fail" assuming sufficiently atomic tests.

anything is provably true given a false assumption:-)

Some tests were not as atomic as perhaps they could have been, but 
sometimes you really need to test things in combination. An 
implementation that can render every single element, but crashes if 
given two elements in the same document isn't a lot of use, so in the 
end you end up wanting to test some elements in combination, and that's 
a slippery slope towards less atomic tests and less definite pass/fail.

David



________________________________________________________________________
The Numerical Algorithms Group Ltd is a company registered in England
and Wales with company number 1249803. The registered office is:
Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.

This e-mail has been scanned for all viruses by Star. The service is
powered by MessageLabs. 
________________________________________________________________________

Received on Tuesday, 30 November 2010 11:12:16 UTC