W3C home > Mailing lists > Public > w3c-wai-ig@w3.org > July to September 2005

Re: Exploding the myth of automated accessibility checking

From: David Poehlman <david.poehlman@handsontechnologeyes.com>
Date: Mon, 8 Aug 2005 21:15:49 -0400
Message-Id: <4F666F29-719E-491E-BD90-210DFAC8D45C@handsontechnologeyes.com>
Cc: "Wendy Chisholm" <wendy@w3.org>, "Joe Clark" <joeclark@joeclark.org>, "WAI-IG" <w3c-wai-ig@w3.org>, "WAI-GL" <w3c-wai-gl@w3.org>
To: Jamal Mazrui <Jamal.Mazrui@fcc.gov>

There is a mind set that would limit the scope of wcag 2.0 to only  
those factors which can be tested by automation.  I would not welcome  
this change because you could then even more than now have a site  
that past but was utterly useless.

Jonnie Apple Seed
With His:
Hands-On Technolog(eye)s

On Aug 8, 2005, at 4:28 PM, Jamal Mazrui wrote:

I think the goal of automated testing, as much as possible, is an
important one.  To me, these results indicate that the success criteria
and testing tools need to be improved, rather than the goal discounted.


-----Original Message-----
From: w3c-wai-ig-request@w3.org [mailto:w3c-wai-ig-request@w3.org] On
Behalf Of Wendy Chisholm
Sent: Monday, August 08, 2005 3:35 PM
To: Joe Clark; WAI-IG; WAI-GL
Subject: Re: Exploding the myth of automated accessibility checking

At 03:13 PM 8/8/2005, Joe Clark wrote:

> National treasure Gez Lemon wrote a test page with known validation  
> and

> WCAG errors and ran it through various automated checking tools, none

> which caught more than a few of the errors, if that.
> <http://juicystudio.com/article/invalid-content-accessibility- 
> validator

Excellent.  This is an important point for people to understand. I
evaluated a Web site last week that had 8 major accessibility issues but

the evaluation tools only found 1 or 2 (depending on the tool).

> It's quite a devastating analysis and calls into question the WCAG

> Group's interest in making as many guidelines as possible

When the WCAG WG talks about testability, our primary goal is to provide

enough information so that people who evaluate or create Web content can

make a good decision.  In WCAG 1.0, some of the checkpoints are
so we're trying to fix that in WCAG 2.0 by providing as much testable
information as possible.  By specifying that success criteria must be
"testable" we are not saying that success criteria are machine
We are saying that a person should be able to determine if they have
satisfied a given criteria. Most of the "tests" (in the test suite) are
procedures for humans to follow, not algorithms for tools. If tests are
automatable, that's great, however I don't think anyone expects that all

tests (or even a majority) will be fully automated.  Shawn Henry  wrote
great piece about this a while ago "Web Accessibility Evaluation Tools
People" [1].

The 30 June 2005 Working Draft of WCAG 2.0 says, "The Working Group
believes that all success criteria should be testable. Tests can be done
computer programs or by people who understand this document. When
people who understand WCAG 2.0 test the same content using the same
criteria, the same results should be obtained."  This could probably use

some work, but I hope that it's clear that we understand humans are part
the evaluation process and that our primary goal is to provide
success criteria.


[1] <http://uiaccess.com/evaltools.html>
Received on Tuesday, 9 August 2005 01:16:07 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 20:36:26 UTC