Re: Discussion: How to weight different accessibility warnings?

On Thu, 7 Feb 2002, Al Gilman wrote:

> [disclaimer -- this was written Tuesday.

And this is a quick partial response to selected/specific points.
>
> The basic principle is that the "overall evaluation" question needs to be
> answered by WCAG, not ER.

Is that wai-wcag-editor?  It doesn't exactly look lively.

> >I've now used five levels:
> > - Certain: we know this violates a guideline; no human check required.
> > - High: A construct that is likely to be wrong, but we're not certain.
> > - Medium: We can't tell; human checking required
> > - Low: Something that's probably OK, but should be flagged for checking.
> > - "-": Messages that definitely don't mean there's a problem.
>
> This last needs better definition. Why is there any event thrown?

There was a very specific reason: pseudo-legalistic completeness in
implementing Section508.  The two messages that carry this weighting are:
  (1) A message in every document, saying "provide a text-only alternative
      if this fails to meet the criteria".
  (2) A message re clientside imagemaps, noting that valid HTML is
      sufficient to meet the guideline.  Since Page Valet is a validator,
      no additional message is required for this section.

> >
> >In producing an overall document score, we simply evaluate the
> >highest confidence warning anywhere in the document:
> >
> > - Certain => Fail
> > - High => Probable Fail - check messages
> > - Medium => Uncertain - check messages carefully!
> > - Low => Probable Pass - check messages
> > - '-' => Pass - no problems found
> >
>
> It would be interesting to set quantitative targets for what the statistics
> would be in an ideal world for these grades.

Indeed it would.  Do we have any volunteers?  I fear if I undertook that
work it might not be seen as an objective assessment.

> But talking about the confidence levels inspires comparison with confidence
> assessment practices in statistics or we sorta need to get quantitative or get
> another word.


>
> >(unconditional pass is very hard indeed, but /WAI/ER/ scores it
> >at WCAG single-A :-)
>
> I beg your pardon? There is no purely machinable way to arrive at a WCAG 1.0
> single-A assertion.

It means the page doesn't trigger any WCAG single-A warnings.  If you
think it should, your comments will be welcome.

> >
> >Now, the Big Issue is assigning priorities. While the basic principle
> >is to describe confidences, that is inevitably often subjective,
> >and I'd really like some feedback on whether people agree with my
> >assignments.
>
> Get Jim Ley to build you a spider and gather some data.

I am running a spider, and have plenty of data.  But again, if the
tool reflects my prejudices (and includes PLACEHOLDER comments denoting
something I already plan to improve on),  then I'm not the right person
to assess it.

> "95% of the hit-weighted web content on the web that flunks this test flunks
> in-depth manual assessment by the experts of the WCAG WG. So please give it
> your careful attention."

Ah, now that's interesting.  Is there any existing corpus that has been
manually assessed?  There's clearly nothing specific to any one assessment
tool in building such a corpus.

> As Kynn has pointed out, a candidate warning nominated by the detection of
> presentational attributes used in the HTML may be entirely pruned away by
> checking for the presence of the CSS to "do it right." That's a higer-level
> rollup that you can do off the logic of the checkpoint itself.

I don't entirely agree with that, and will get round to responding at
some point (if noone helps out with a followup I find easier to work with).

-- 
Nick Kew

Site Valet - the mark of Quality on the Web.
<URL:http://valet.webthing.com/>

Received on Thursday, 7 February 2002 19:50:34 UTC