Long Re: A question of interpretation

My opinion in short:

Use whatever tools it makes sense to use, and be prepared to justify 
the results according to what you are testing. (I am not anti-tools - I 
think they are great. The more you know about how your tool works and 
what it is good and bad at the better it can help you...)

Testing conformance to the guidelines means testing against what they 
say. If you want to write a specific policy for using the guidelines, 
go ahead, but don't mistake that for the guidelines. You can tell if 
they change by checking the authorative version on the W3C website, and 
so far the recommendation is exactly as it was in mid-may 1999.

More detailed answers to some things that seem obvious, and notes about 
things I cannot answer, interspersed below.

On Tuesday, Sep 9, 2003, at 12:07 Australia/Sydney, tina@greytower.net 
wrote:

>    Funka Nu's comment:
>     The test method used does not follow WAI's recommended procedure
>     for a website audit. Among other thing a proprietary, non-free
>     product has been used for the analysis. A government department,
>     or other independent reviewer, can not repeat the analysis to
>     compare the results. Several goverment departments have had
>     difficulty obtaining the test results when questioning the
>     report.
>
>    Our answer:
>     The methodology used is, as described in our article, that of the
>     WAI Preliminary Review. We have changed it only in response to two
>     facts: all testing could not be done online, and testing needed to
>     be done speedily. There were 92 different websites to test, and 
> only
>     22 days in which to test them. We accept that the way we described
>     the methodology may have lead someone to believe that a method of
>     our own invention were in use.
>
>     However: we do indeed use a proprietary, non-free, tool for our
>     testing. It is an inhouse system called siteSifter. It is our
>     impression that the WAI testing guidelines does not specifically
>     *demand* that one of the tools on the list be used for the
>     methodology to be valid. Is this impression correct ?

CMN
The WAI testing methodology is not a document endorsed by W3C. It is 
some ideas published by a working group, who have not even yet 
published it as a formal W3C draft.

Tina...
>     If using a tools listed on
>
>       http://www.w3.org/WAI/ER/existingtools.html#General
>
>     is a requirement, which process has been used for selecting just
>     these tools ? Is there a review tool testing system in place that
>     ensures tool compliance ? If so, where ? I believe that several of
>     those listed are also proprietary and non-free.
CMN
There is no review system I know of for putting tools onto that list - 
when I was working on the list the process was the entirely random one 
of "if I hear of a tool I figure it would be good to add it". Wendy 
Chisholm may be able to confirm whether or not this is still the 
process used.
Tina
>     Is it really the intent of the W3C WAI that an organization should
>     purchase a specific tool - be it Bobby, siteSifter, or Sitevalet - 
> in
>     order to comply with the guidelines ?
CMN
W3C is a vendor-neutral organisation. It doesn't actually insist on 
people using the W3C HTML validator - there are two other HTML 
validators linked directly from the W3C validator home page, and there 
are others around.

>    Funka Nu's comment:
>     The Audit Office has interpreted all the checkpoints literally, 
> and as
>     if these had equal value. No concessions have been made for 
> conditions
>     specific to Sweden, or the development of assistive technology 
> since
>     the guidelines were written. If a flaw has been found that goes 
> against
>     a point in WAI, this flaw has automatically led to the checkpoint 
> being
>     marked as "failed", whether or not the flaw is serious or not.
>
>    Our answer:
>     This, to us, is the most puzzling of the critique. The WCAG 1.0 
> checkpoints
>     are divided into three sets, priority 1, 2 and 3. As such they were
>     tested, and as such they have been analyzed and reported. This is 
> clear
>     from the actual report, where results for each priority has been 
> separately
>     listed.
>
>     However: the question raised is one of re-interpretation and 
> adjustment
>     of guidelines. It is argued that since, in Sweden, all visually 
> impaired
>     users are given modern, well-updated, equipment[2] of the same 
> type[3],
>     there is no point in testing for other equipment. Nor is there any 
> need
>     to correct for these "other" systems.
>
>     An argument is made that doing so will remove focus from the 
> actual,
>     important, accessibility work.
>
>     Again, this raises another level of questions. It is, of course, 
> not
>     possible for the WAI-IG list to comment on the intent of the 
> European
>     Union, but I can personally not find any sign that their adoption 
> of
>     the WCAG 1.0[4] include any leeway for interpretation based on 
> perceived
>     national differences. It does, quite cathegorically, state that:
>
>       "The Guidelines aim to be compatible both with earlier 
> technologies
>        and Web design tools and also with new technologies and tools, 
> for
>        example, with new types of Web browsers such as digital 
> assistants
>        and WAP telephones." - COM(2001) 529
>
>     leading me to conclude that the European Union, for one, does not 
> plan
>     on "dissing" older technology, nor refuse to adopt new.
>
>     When analysing the WAI and WCAG itself, including the section 
> marked
>     as "Considerations for Specific Contexts", I can find no mention 
> that
>     the guidelines are meant to be adjusted for region-specific 
> cultural,
>     political, or even technical details. Is my interpretation correct 
> ?
>
>     I can personally see no meaning in applying the WCAG for use, for
>     instance in the EU, if the guidelines are to be re-interpreted for 
> each
>     country or region. It is clear that there are parts of the Union, 
> as
>     there are parts of the world, where the level of technology may 
> not be
>     as "high" as in Sweden, but that - in my view - was part of the 
> point
>     of the accessibility guidelines.
>
>     Have I misunderstood ? A conformance to a certain priority is one
>     thing, but conforming to a certain, perceived, level of technology
>     would seen to go entirely against the principles of accessibility.
CMN
My understanding is that the Guidelines are meant to describe a set of 
requirements, and testing against the guidelines should result in 
statements that something does or does not conform to the guidelines. 
This is the approach I am taking to my work in the EuroAccessibility 
Consortium, and in other situations.

There are cases where I have suggested developing a policy that 
explicitly states what WCAG or other results are required. For example, 
in an entirely intranet situation I have suggested that certain 
checkpoints need not be met, while other checkpoints should be treated 
as a higher priority than the guidelines assign them. This  means that 
testing for such a situation requires testing against WCAG _and_ 
testing the results against the policy described. But a test against 
WCAG is just that. Interpretation of the "until user agents..." 
checkpoints is very difficult in the absence of any response from the 
working group on how to make that judgement call, and I have in one 
recent case had to go back on my earlier assumption about what user 
agents now do. But the guidelines do not say "feel free to reinterpret 
checkpoints according to local custom..." - they are meant to be 
testable and to produce consistent results for tests, which is 
impossible if people simply decide to ignore a bit through local custom.

Tina...
>   Finally, claims are being made - publically - in Sweden that
>  The Swedish Agency for Public Management[5], The Swedish Disability 
> Ombudsman,
>  and the Swedish W3C Office are enganged in a joint project to 
> reinterpret
>  the WCAG for Swedish "realities".
>
>   This, frankly, worries me. The claim has been made in seminars by 
> people
>  who "ought to know". I would like to invite comment on this from 
> WAI-IG
>  members, and in particular from W3C staff.
CMN
The Swedish W3C Office is independent from WAI - it is a W3C member who 
meets certain conditions and takes on some work on behalf of W3C. I am 
not a WAI staff member, so can't speak for WAI (and as it happens I 
wouldn't know if they were involved in something like this or not - my 
staff role at W3C keeps me busy doing my own work ;-). It is possible 
that these three groups are trying to write a specific policy for using 
WAI work. If so, they would be the people to ask.

I don't know exactly what WAI's position is on doing this - Judy Brewer 
was involved in the committee set up to advise on rules for section 508 
(although the recommendations of that committee were largely ignored in 
the rule-making) and WAI clearly has an interest. I know that in 
general they have spoken against the "fragmentation" of accessibility 
policy, and when I worked for them on authoring tools it was clear that 
different policies in different areas were not going to help 
manufacturers implement compliant systems.

These are simply my own opinions on how to interpret WCAG and W3C work, 
based on the questions asked. Naturally I offer no comment on the facts 
of the case in Sweden - all the information I have on that is what Tina 
has sent to the list.

cheers

Chaals
--
Charles McCathieNevile                          Fundación Sidar
charles@sidar.org                                http://www.sidar.org

Received on Monday, 8 September 2003 23:23:45 UTC