W3C home > Mailing lists > Public > www-qa@w3.org > September 2003

Re: browsable test results

From: <david_marston@us.ibm.com>
Date: Fri, 12 Sep 2003 13:02:26 -0400
To: www-qa@w3.org
Cc: w3c-wai-er-ig@w3.org
Message-ID: <OF0F53DA27.8A46D027-ON85256D9F.005C9E93@lotus.com>

Patrick Curran responded to my earlier posting:
DM>Implication that the current suite is 100% of all the tests that
DM>should be there. Test suites that are being expanded frequently
DM>won't have a stable notion of 100%.

PC>Isn't there a more fundamental problem? Test suites should be
PC>versioned. It ought to be OK to state "I passed x% of the test cases
PC>for version y.z of the test suite." The number of test cases in any
PC>particular version of the test suite should be, of course, fixed.

Sure. That's just one of many assumptions I didn't bother to say in my
reply. I have a history of supporting versioning for test suites that
goes back to remarks I made at the W3C QA Workshop in 2001. Further,
a test lab should be able to obtain either (1) all the approved tests
for the current or any prior numbered version, or (2) the current set
of both approved and proposed test cases. Use (2) to test the state of
the test suite and/or harness.

When a vendor or test lab wishes to report on the results of testing a
product against a W3C-approved test suite, there will be more verbiage
expected in the formal claim. In particular, for many specs no product
will be subjected to 100% of the approved tests, just the set of all
tests that fit that product's choices on the Dimensions of Variability.
.................David Marston
Received on Friday, 12 September 2003 13:02:58 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:14:00 GMT