W3C home > Mailing lists > Public > w3c-wai-ig@w3.org > July to September 2005

Re: Exploding the myth of automated accessibility checking

From: Wendy Chisholm <wendy@w3.org>
Date: Mon, 08 Aug 2005 15:34:33 -0400
Message-Id: <>
To: Joe Clark <joeclark@joeclark.org>, WAI-IG <w3c-wai-ig@w3.org>, WAI-GL <w3c-wai-gl@w3.org>

At 03:13 PM 8/8/2005, Joe Clark wrote:

>National treasure Gez Lemon wrote a test page with known validation and 
>WCAG errors and ran it through various automated checking tools, none of 
>which caught more than a few of the errors, if that.

Excellent.  This is an important point for people to understand. I 
evaluated a Web site last week that had 8 major accessibility issues but 
the evaluation tools only found 1 or 2 (depending on the tool).

>It's quite a devastating analysis and calls into question the WCAG Working 
>Group's interest in making as many guidelines as possible machine-checkable.

When the WCAG WG talks about testability, our primary goal is to provide 
enough information so that people who evaluate or create Web content can 
make a good decision.  In WCAG 1.0, some of the checkpoints are ambiguous, 
so we're trying to fix that in WCAG 2.0 by providing as much testable 
information as possible.  By specifying that success criteria must be 
"testable" we are not saying that success criteria are machine automatable. 
We are saying that a person should be able to determine if they have 
satisfied a given criteria. Most of the "tests" (in the test suite) are 
procedures for humans to follow, not algorithms for tools. If tests are 
automatable, that's great, however I don't think anyone expects that all 
tests (or even a majority) will be fully automated.  Shawn Henry  wrote a 
great piece about this a while ago "Web Accessibility Evaluation Tools Need 
People" [1].

The 30 June 2005 Working Draft of WCAG 2.0 says, "The Working Group 
believes that all success criteria should be testable. Tests can be done by 
computer programs or by people who understand this document. When multiple 
people who understand WCAG 2.0 test the same content using the same success 
criteria, the same results should be obtained."  This could probably use 
some work, but I hope that it's clear that we understand humans are part of 
the evaluation process and that our primary goal is to provide unambiguous 
success criteria.


[1] <http://uiaccess.com/evaltools.html>
Received on Monday, 8 August 2005 19:34:42 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 20:36:26 UTC