Re: Proposal: severity axis on test result

Hmm. This was similar to somethingthe UAAG group wanted.

I would propose that we have qualitative rather than quantitative ratings.
For some use cases you Yb - pass with unrelated errors would count as a pass,
and for some cases it would score as a fail. So we would need to know what
they are.

The question also arises as to how many kinds of result we should include in
earl and at what point we should leave people to subclass them for their own
more detailed uses.

Cheers

Chaals

On Tue, 25 Jun 2002, Ian Hickson wrote:


  Test results currently have a type (pass, fail, not applicable, etc) and a
  confidence (high, medium, low).

  I propose you add a third property: severity. This would be a record of how well
  or badly the test passed or failed.

  For example, a layout test could:

      pass completely                        pass, severity 100%    Y
      pass with unrelated errors             pass, severity 90%     Yb
      pass partially                         pass, severity 50%     P
      not be implemented at all              fail, severity 0%      N
      fail                                   fail, severity 50%     B
      fail so badly the feature is unusable  fail, severity 90%     D
      crash                                  fail, severity 100%    C

  This is very important for tests of features that can be implemented to
  different degrees. For example, CSS tests need this kind of report. You can see
  in the following test results page how this is used (except instead of severity,
  I used the codes on the right hand side of the column):

  http://www.hixie.ch/tests/tesremas/listresults.pl?ID=ETS&mainSortH=Tests&mainSortV=Score&mainMinTests=20&mainTrimmed=on

  This is separate from "importance", which is a property of the test itself.

  Cheers,


-- 
Charles McCathieNevile    http://www.w3.org/People/Charles  phone: +61 409 134 136
W3C Web Accessibility Initiative     http://www.w3.org/WAI  fax: +33 4 92 38 78 22
Location: 21 Mitchell street FOOTSCRAY Vic 3011, Australia
(or W3C INRIA, Route des Lucioles, BP 93, 06902 Sophia Antipolis Cedex, France)

Received on Sunday, 30 June 2002 18:23:46 UTC