Re: browsable test results

Good point. "100%" should be interpreted as "100% of the tests that are 
appropriate for my implementation, given the DOVs that I implement"...

david_marston@us.ibm.com wrote:

>Patrick Curran responded to my earlier posting:
>DM>Implication that the current suite is 100% of all the tests that
>DM>should be there. Test suites that are being expanded frequently
>DM>won't have a stable notion of 100%.
>
>PC>Isn't there a more fundamental problem? Test suites should be
>PC>versioned. It ought to be OK to state "I passed x% of the test cases
>PC>for version y.z of the test suite." The number of test cases in any
>PC>particular version of the test suite should be, of course, fixed.
>
>Sure. That's just one of many assumptions I didn't bother to say in my
>reply. I have a history of supporting versioning for test suites that
>goes back to remarks I made at the W3C QA Workshop in 2001. Further,
>a test lab should be able to obtain either (1) all the approved tests
>for the current or any prior numbered version, or (2) the current set
>of both approved and proposed test cases. Use (2) to test the state of
>the test suite and/or harness.
>
>When a vendor or test lab wishes to report on the results of testing a
>product against a W3C-approved test suite, there will be more verbiage
>expected in the formal claim. In particular, for many specs no product
>will be subjected to 100% of the approved tests, just the set of all
>tests that fit that product's choices on the Dimensions of Variability.
>.................David Marston
>
>  
>

Received on Friday, 12 September 2003 19:23:31 UTC