Dan Connolly wrote:
...How much of the wheel have we reinvented?
From the point of view of the OASIS test case management system used
for XSLT, not too much. At a quick glance, I think the two are
compatible.
From the point of view of QAWG guidelines, I see a problem in the
result table where you report that a product has passed 100% of the
test cases for each group. The string "No failures found" would be
more appropriate. The issues about percentages include:
1. Implication that the current suite is 100% of all the tests that
should be there. Test suites that are being expanded frequently
won't have a stable notion of 100%.
Isn't there a more fundamental problem? Test suites should be
versioned. It ought to be OK to state "I passed x% of the test cases
for version y.z of the test suite." The number of test cases in any
particular version of the test suite should be, of course, fixed.
2. Implication that high numbers under 100% are pretty good. Each
class of product may have its own notion of how seriously interop
has been hurt by a score of, say, 96%.
3. Implication that all tests count equally. Product X's 96% might
be much worse than Product Y's 96%, depending on the cases that
comprise the failing 4% on each.
.................David Marston
Right. Attempting to interpret any claim other than "I passed all the
tests" is a dangerous business. It can have value for the implementors,
and help them to figure out where they need to improve, but should not
be used to compare one implementation with another...