W3C home > Mailing lists > Public > www-qa@w3.org > September 2003

Re: browsable test results

From: <david_marston@us.ibm.com>
Date: Fri, 12 Sep 2003 10:52:49 -0400
To: www-qa@w3.org
Cc: w3c-wai-er-ig@w3.org
Message-ID: <OF01F27187.5DB2501C-ON85256D9F.0050C76E@lotus.com>

Dan Connolly wrote:
>...How much of the wheel have we reinvented?

From the point of view of the OASIS test case management system used
for XSLT, not too much. At a quick glance, I think the two are
compatible.

From the point of view of QAWG guidelines, I see a problem in the
result table where you report that a product has passed 100% of the
test cases for each group. The string "No failures found" would be
more appropriate. The issues about percentages include:
1. Implication that the current suite is 100% of all the tests that
   should be there. Test suites that are being expanded frequently
   won't have a stable notion of 100%.
2. Implication that high numbers under 100% are pretty good. Each
   class of product may have its own notion of how seriously interop
   has been hurt by a score of, say, 96%.
3. Implication that all tests count equally. Product X's 96% might
   be much worse than Product Y's 96%, depending on the cases that
   comprise the failing 4% on each.
.................David Marston
Received on Friday, 12 September 2003 10:52:59 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:14:00 GMT