Discussion: How to weight different accessibility warnings? (fwd)

This was first posted to w3c-wai-er-ig, where it has been suggested
that this forum would be more appropriate.

A supplementary question arising in a followup was whether any work
has been done on creating a test corpus, which might indicate that
 "Construct X has a [nn]% chance of giving rise to accessibility problem Y"
and that I could use.


---------- Forwarded message ----------
From: Nick Kew <nick@webthing.com>
To: w3c-wai-er-ig@w3.org
Subject: Discussion: How to weight different accessibility warnings?
Date: Wed, 6 Feb 2002 01:49:04 +0000 (GMT)


Page Valet now offers fairly comprehensive page evaluation against
the WCAG and US Section 508 accessibility guidelines.

I'm now working through the issues of

(1) distinguishing errors from warnings , and
(2) assigning an overall evaluation to a document

To do so, I've established a set of confidence levels, and assigned
one to each test.  This is in principle orthogonal to the WCAG
priorities, and should measure how likely Page Valet thinks it is
that a guideline has in fact been breached:

e.g. - a Frame without a title is clearly a breach, so we can flag it
       with high confidence.
     - <strong>This text is emphasised</strong> might possibly be a
       header, so we query whether it should be.  But the chances are
       it's being correctly used, so this is a low-confidence warning.

I've now used five levels:
  - Certain: we know this violates a guideline; no human check required.
  - High: A construct that is likely to be wrong, but we're not certain.
  - Medium: We can't tell; human checking required
  - Low: Something that's probably OK, but should be flagged for checking.
  - "-": Messages that definitely don't mean there's a problem.

In producing an overall document score, we simply evaluate the
highest confidence warning anywhere in the document:

  - Certain	=>	Fail
  - High	=>	Probable Fail - check messages
  - Medium	=>	Uncertain - check messages carefully!
  - Low		=>	Probable Pass - check messages
  - '-'		=>	Pass - no problems found

(unconditional pass is very hard indeed, but /WAI/ER/ scores it
at WCAG single-A :-)


Now, the Big Issue is assigning priorities.  While the basic principle
is to describe confidences, that is inevitably often subjective,
and I'd really like some feedback on whether people agree with my
assignments.  I should add that I have made some conscious decisions to
stray from the True Path of Confidence, in deference to real-world
considerations.  For example, presentational HTML will generate
a message "Use CSS for layout and presentation" at WCAG-AA or higher.
(http://www.w3.org/TR/WCAG10/#tech-style-sheets), but the "border"
attribute is low-confidence (IMO it's not really harmful and it
does have legit. uses as a browser workaround) while other
presentational things will generate higher-confidence warnings.

Please folks, play with it, and let me know if you think my
confidence levels make sense!

-- 
Nick Kew

Site Valet - the mark of Quality on the Web.
<URL:http://valet.webthing.com/>

Received on Monday, 11 February 2002 18:20:57 UTC