W3C home > Mailing lists > Public > www-qa@w3.org > September 2003

Re: browsable test results

From: Patrick Curran <Patrick.Curran@Sun.COM>
Date: Fri, 12 Sep 2003 16:20:34 -0700
To: david_marston@us.ibm.com
Cc: www-qa@w3.org, w3c-wai-er-ig@w3.org
Message-id: <3F6254C2.5000309@sun.com>

Good point. "100%" should be interpreted as "100% of the tests that are 
appropriate for my implementation, given the DOVs that I implement"...

david_marston@us.ibm.com wrote:

>Patrick Curran responded to my earlier posting:
>DM>Implication that the current suite is 100% of all the tests that
>DM>should be there. Test suites that are being expanded frequently
>DM>won't have a stable notion of 100%.
>
>PC>Isn't there a more fundamental problem? Test suites should be
>PC>versioned. It ought to be OK to state "I passed x% of the test cases
>PC>for version y.z of the test suite." The number of test cases in any
>PC>particular version of the test suite should be, of course, fixed.
>
>Sure. That's just one of many assumptions I didn't bother to say in my
>reply. I have a history of supporting versioning for test suites that
>goes back to remarks I made at the W3C QA Workshop in 2001. Further,
>a test lab should be able to obtain either (1) all the approved tests
>for the current or any prior numbered version, or (2) the current set
>of both approved and proposed test cases. Use (2) to test the state of
>the test suite and/or harness.
>
>When a vendor or test lab wishes to report on the results of testing a
>product against a W3C-approved test suite, there will be more verbiage
>expected in the formal claim. In particular, for many specs no product
>will be subjected to 100% of the approved tests, just the set of all
>tests that fit that product's choices on the Dimensions of Variability.
>.................David Marston
>
>  
>
Received on Friday, 12 September 2003 19:23:31 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:14:00 GMT