Proposal: severity axis on test result

Test results currently have a type (pass, fail, not applicable, etc) and a 
confidence (high, medium, low).

I propose you add a third property: severity. This would be a record of how well 
or badly the test passed or failed.

For example, a layout test could:

    pass completely                        pass, severity 100%    Y
    pass with unrelated errors             pass, severity 90%     Yb
    pass partially                         pass, severity 50%     P
    not be implemented at all              fail, severity 0%      N
    fail                                   fail, severity 50%     B
    fail so badly the feature is unusable  fail, severity 90%     D
    crash                                  fail, severity 100%    C

This is very important for tests of features that can be implemented to 
different degrees. For example, CSS tests need this kind of report. You can see 
in the following test results page how this is used (except instead of severity, 
I used the codes on the right hand side of the column):

http://www.hixie.ch/tests/tesremas/listresults.pl?ID=ETS&mainSortH=Tests&mainSortV=Score&mainMinTests=20&mainTrimmed=on

This is separate from "importance", which is a property of the test itself.

Cheers,
-- 
Ian Hickson                                      )\._.,--....,'``.    fL
"meow"                                          /,   _.. \   _\  ;`._ ,.
http://index.hixie.ch/                         `._.-(,_..'--(,_..'`-.;.'

Received on Tuesday, 25 June 2002 12:00:27 UTC