Re: browsable test results

Patrick Curran responded to my earlier posting:
DM>Implication that the current suite is 100% of all the tests that
DM>should be there. Test suites that are being expanded frequently
DM>won't have a stable notion of 100%.

PC>Isn't there a more fundamental problem? Test suites should be
PC>versioned. It ought to be OK to state "I passed x% of the test cases
PC>for version y.z of the test suite." The number of test cases in any
PC>particular version of the test suite should be, of course, fixed.

Sure. That's just one of many assumptions I didn't bother to say in my
reply. I have a history of supporting versioning for test suites that
goes back to remarks I made at the W3C QA Workshop in 2001. Further,
a test lab should be able to obtain either (1) all the approved tests
for the current or any prior numbered version, or (2) the current set
of both approved and proposed test cases. Use (2) to test the state of
the test suite and/or harness.

When a vendor or test lab wishes to report on the results of testing a
product against a W3C-approved test suite, there will be more verbiage
expected in the formal claim. In particular, for many specs no product
will be subjected to 100% of the approved tests, just the set of all
tests that fit that product's choices on the Dimensions of Variability.
.................David Marston

Received on Friday, 12 September 2003 13:02:58 UTC