Re: What happens when an evaluation statement already exists?

Alistair:

> I have before read in this groups responses to my emails a note of
 > seeming disregard for the skills and honesty of web developers
 > / website owners, and I sadly read it again here.

*Many* owners / developers are skilled and honest, but *not all* owners 
/ developers are. In my experience, many owners just want a label to 
avoid legal issues, and many developers don't have the skills, or they 
(or the project managers) don't want to make the effort needed to make 
certain things accessible.

And, of course, I am never saying that they are bad persons, indeed I 
simply think that many of them have their own priorities and that some 
see accessibility as a bug more than a feature. In any case, I am also a 
developer, so I know the problems that we have to solve sometimes, and 
my bosses/clients are not always happy spending the time / money that 
some accessible developments need.

Assuming that all owners are skilled and trustful is as bad, or even 
worse, as assuming that all of them are ignorant and evil.


 > I personally think we need to trust people when they make a public
 > statement, which is why I made my proposal for an addition to
 > step 1.d:

I can trust them in their intention, but maybe they don't have the 
knowledge. And if they have the knowledge, why should they be afraid of 
my evaluation? Why should I be forced to use *their* evaluation 
procedure instead of mine? If they are really conforming, I will not be 
able to find any failure. Or is it that you don't trust *me* when I 
evaluate *their* content? If we must respect and trust owners / 
developers, why not respect and trust evaluators?


> I should like to state clearly that the WCAG 2.0 guidelines are
 > developed for web developers, rather than evaluators.

Maybe, but according to the spec, "WCAG 2.0 success criteria are written 
as testable statements", which means, in my opinion, that the success 
criteria are written -also- with evaluation in mind. The aim of WCAG 2.0 
itself is to provide guidance to create accessible content, but also 
ensuring that everyone is able to verify if web pages follow the Success 
Criteria (and meet the Conformance Requirements).


 > I should also like to state that the evaluation methodology we
 > have created is for use by first and third parties i.e. web
 > developers and evaluators.

I don't see how this affects the subject. A developer can use the 
methodology, but an external/independent researcher can use it, too, 
following a different evaluation procedure. Why should we impose one 
evaluation procedure over the other? Why trust the developer and not the 
external evaluator?


> It is also important to underline that expert evaluators are very
 > clearly not the only people to be able to say whether or not
 > something is accessible, especially now that WCAG 2.0's sufficient
 > techniques exist.

Maybe, but so-called "sufficient" techniques are only sufficient to 
Conform when they are accessibility supported. Many many many developers 
treat sufficient techniques as -normative- Golden Rules, without taking 
into account if they are accessibility support or not. Conforming 
alternate versions are commonly misunderstood, too, as if they can be 
used to bypass WCAG as a whole, just putting a link to the "text 
version" and then claiming WCAG conformance for the main version.

Expert evaluation is not needed, but understading of WCAG Conformance is 
a must.

Regards,
Ramón.

 >> Previously...
 >>
>> Alistair wrote:
>>
>>
>>> My core question is "Have you ever created a web page and tested
>>> it with different tools?  Did you get the same results?"
>> Yes, of course, but I doubt that a tool that claims itself to be WCAG compliant can lead to false positives or, if you prefer, if the tool reports that a conforming website is not meeting WCAG, then the tool should be discarded. If a tool is WCAG compliant, it should not detect any failure in a conforming website. Maybe the tool includes warnings for manual checks, but it is unlikely that a tool will say "the table is wrong because you didn't use @scope", it will probably say "I cannot find @scope, so check if the data cells are associated somehow to the appropriate header cells".
>>
>> Yes, maybe today a tool says "the table is wrong because there are no <th> elements", but you used roles and other ARIA attributes to emulate the table structure, but then it is the tool what needs an update, not the website, and your claim is perfectly valid. If a tool states that a particular content is wrong, but this particular content is using a documented and accessibility supported technique, it is easy to claim that the tool has a flaw, and not your website
>>
>> Therefore, I don't see the point of imposing a precedence of evaluation techniques (again, not developing techniques). As evaluator, I think that it is more important to ensure that the assessment does not depend on the *evaluation* techniques used by the owner, because it is possible that the owner is using a restrictive set of techniques just to bypass real conformance to WCAG (for example, using techniques that are not accessibility supported).
>>
>>
>>> All I am saying is that respect should be paid to the
>>> WCAG techniques, and their directly associated checks,
>>> a developer implements in order to make their web pages
>>> accessible.
>> Techniques are not normative. I think this sentence suggests that they can be used as normative rules for evaluating conformance, with precedence over the Conformance Requirements, that could be tested using different techniques.
>>
>>
>>> If not, you will get the situation where a single wide-scale
>>> monitoring tool is developed which tests certain specific things,
>>> and in order to comply all web developers naturally start to make
>>> their web pages accessible in exactly the same way.  Which is
>>> not what WCAG is about...
>> I guess that risk is even greater if website oweners can force the techniques that must be used to an independent evaluator. Website owners might indeed focus their efforts to pass the tests from a specific tool, and then claim Conformance based only on the results of that tool. But maybe the tool use only a restrictive set of tests that do not cover every aspect of WCAG, or that do not take into account accessibility support, for example, or that do not recognise processes, etc.
>>
>> So my vote is still for allowing evaluators to use any methods that they consider, but obviously they must use methods that ensure WCAG conformance, not "MyTool-Conformance". Maybe I am not understading your point, but I cannot imagine a situation where an independent analysis is not desirable.
>>
>> My mainm concern is that your proposal...
>>
>> "If documented evaluation procedures are provided along with an evaluation statement or WCAG 2.0 conformance claim, the documented evaluation procedures provided should take precedence over any other evaluation procedure when undertaking an evaluation."
>>
>>
>> ... may also be read this way: "I am a website owner that used "MyTool", that is supposed to be a complete tool to automatically assess WCAG 2.0 conformance. "MyTool" told me that my website is conforming to Double-A level, and therefore I made a claim stating that my website meets AA. I don't mind if you used your own procedures and discovered that my website is not keyboard accessible or that the descriptions of my images are meaningless, because my documented evaluation procedure ("I used MyTool") must take precedence over yours."
>>
>> Regards,
>> Ramón.
>>
>>
>>>> Ramón said: 
>>>>
>>>> From the "Techniques for WCAG 2.0" document:
>>>>
>>>> "Please note that the contents of this document are informative (they provide guidance), and not normative (they do not set requirements for conforming to WCAG 2.0)."
>>>>
>>>> The only normative document is the WCAG 2.0 Recommendation, so you cannot tell anyone that you have to use a particular technique to check the accessibility of a website. Techniques may -and will- change in the future due to new technologies or methods, varying accessibility support, wrong concepts or any other circumstance.
>>>>
>>>> Of course, any WCAG-related evaluation procedure must ensure that WCAG 2.0 itself is met, taking into account that conformance can be reached through many different ways, provided that they are accessibility supported; indeed, WCAG Conformance comes from meeting the 5 Conformance Requirements, and the Success Criteria are indirectly pointed from them.
>>>>
>>>> If a website owner claims conformance and the website is in fact WAG 2.0 conformant, the method used to assess this should be irrelevant. If a particular method to assess WCAG 2.0 conformance states that a conformant website is non-conformant, there is a flaw in the method, and not in the website or in WCAG.
>>>>
>>>> That said, it is true that sometimes we may disagree about the interpretation of certain important things, such as the required degree of accessibility support, the applicability of conforming alternate versions or other things that can lead to different results.
>>>>
>>>> Of course, these differences can be very important from a legal perspective, but I assume that WCAG 2.0 is a technical document, not a legal one, and if someone cannot access and a discrimination exists, it doesn't matter if you claim conformance, and you cannot excuse the discrimination because "technically" you meet the guidelines.
>>>>
>>>> In any case, overimposing a precedence on the techniques used to evaluate (not the techniques used to develop) could lead to owners claiming conformance but not conforming to WCAG 2.0, and then pretending that you must use *their* techniques to evaluate, not yours. For example, they could say that they used the keyboard (without a screen reader) to check keyboard accessibility, but if I check with the screen reader running maybe the website is completely unusable. Should I agree with the claim just because they did not test a real world case? Must I consider this website accessible?
>>>>
>>>> In conclusion, I am totally against any imposition that gives more precedence to a particular evaluation procedure, of course provided that the evaluation methods follow the WCAG Recommendation.
>>>>
>>>> Regards,
>>>> Ramón.
>>>>
>>>>
>>>> Alistair said:
>>>>
>>>>> If I say I have used this set of techniques, and I have undertaken all associated checks, I would not want to be told that I did not conform because I did not use another set of techniques - possibly bound together in some monitoring tool.  Or, think about two EU states using slightly different monitoring tools - which conflict.   Having chaired the Evaluation Methods Task Force for the EuroAccessibility Consortium, and a similar European task force in the initial WABCluster I can say that such conflicts are the rule, rather than the exception. So, a concrete proposal for an addition to step 1.d would be:
>>>>> "If documented evaluation procedures are provided along with an evaluation statement or WCAG 2.0 conformance claim, the documented evaluation procedures provided should take precedence over any other evaluation procedure when undertaking an evaluation."
>

Received on Tuesday, 20 May 2014 12:22:44 UTC