Re: What happens when an evaluation statement already exists?

Alistair wrote:


> My core question is "Have you ever created a web page and tested
 > it with different tools?  Did you get the same results?"

Yes, of course, but I doubt that a tool that claims itself to be WCAG 
compliant can lead to false positives or, if you prefer, if the tool 
reports that a conforming website is not meeting WCAG, then the tool 
should be discarded. If a tool is WCAG compliant, it should not detect 
any failure in a conforming website. Maybe the tool includes warnings 
for manual checks, but it is unlikely that a tool will say "the table is 
wrong because you didn't use @scope", it will probably say "I cannot 
find @scope, so check if the data cells are associated somehow to the 
appropriate header cells".

Yes, maybe today a tool says "the table is wrong because there are no 
<th> elements", but you used roles and other ARIA attributes to emulate 
the table structure, but then it is the tool what needs an update, not 
the website, and your claim is perfectly valid. If a tool states that a 
particular content is wrong, but this particular content is using a 
documented and accessibility supported technique, it is easy to claim 
that the tool has a flaw, and not your website

Therefore, I don't see the point of imposing a precedence of evaluation 
techniques (again, not developing techniques). As evaluator, I think 
that it is more important to ensure that the assessment does not depend 
on the *evaluation* techniques used by the owner, because it is possible 
that the owner is using a restrictive set of techniques just to bypass 
real conformance to WCAG (for example, using techniques that are not 
accessibility supported).


 > All I am saying is that respect should be paid to the
 > WCAG techniques, and their directly associated checks,
 > a developer implements in order to make their web pages
 > accessible.

Techniques are not normative. I think this sentence suggests that they 
can be used as normative rules for evaluating conformance, with 
precedence over the Conformance Requirements, that could be tested using 
different techniques.


 > If not, you will get the situation where a single wide-scale
 > monitoring tool is developed which tests certain specific things,
 > and in order to comply all web developers naturally start to make
 > their web pages accessible in exactly the same way.  Which is
 > not what WCAG is about...

I guess that risk is even greater if website oweners can force the 
techniques that must be used to an independent evaluator. Website owners 
might indeed focus their efforts to pass the tests from a specific tool, 
and then claim Conformance based only on the results of that tool. But 
maybe the tool use only a restrictive set of tests that do not cover 
every aspect of WCAG, or that do not take into account accessibility 
support, for example, or that do not recognise processes, etc.

So my vote is still for allowing evaluators to use any methods that they 
consider, but obviously they must use methods that ensure WCAG 
conformance, not "MyTool-Conformance". Maybe I am not understading your 
point, but I cannot imagine a situation where an independent analysis is 
not desirable.

My mainm concern is that your proposal...

"If documented evaluation procedures are provided along with an 
evaluation statement or WCAG 2.0 conformance claim, the documented 
evaluation procedures provided should take precedence over any other 
evaluation procedure when undertaking an evaluation."


... may also be read this way: "I am a website owner that used "MyTool", 
that is supposed to be a complete tool to automatically assess WCAG 2.0 
conformance. "MyTool" told me that my website is conforming to Double-A 
level, and therefore I made a claim stating that my website meets AA. I 
don't mind if you used your own procedures and discovered that my 
website is not keyboard accessible or that the descriptions of my images 
are meaningless, because my documented evaluation procedure ("I used 
MyTool") must take precedence over yours."

Regards,
Ramón.


>> Ramón said: 
 >>
>> From the "Techniques for WCAG 2.0" document:
>>
>> "Please note that the contents of this document are informative (they provide guidance), and not normative (they do not set requirements for conforming to WCAG 2.0)."
>>
>> The only normative document is the WCAG 2.0 Recommendation, so you cannot tell anyone that you have to use a particular technique to check the accessibility of a website. Techniques may -and will- change in the future due to new technologies or methods, varying accessibility support, wrong concepts or any other circumstance.
>>
>> Of course, any WCAG-related evaluation procedure must ensure that WCAG 2.0 itself is met, taking into account that conformance can be reached through many different ways, provided that they are accessibility supported; indeed, WCAG Conformance comes from meeting the 5 Conformance Requirements, and the Success Criteria are indirectly pointed from them.
>>
>> If a website owner claims conformance and the website is in fact WAG 2.0 conformant, the method used to assess this should be irrelevant. If a particular method to assess WCAG 2.0 conformance states that a conformant website is non-conformant, there is a flaw in the method, and not in the website or in WCAG.
>>
>> That said, it is true that sometimes we may disagree about the interpretation of certain important things, such as the required degree of accessibility support, the applicability of conforming alternate versions or other things that can lead to different results.
>>
>> Of course, these differences can be very important from a legal perspective, but I assume that WCAG 2.0 is a technical document, not a legal one, and if someone cannot access and a discrimination exists, it doesn't matter if you claim conformance, and you cannot excuse the discrimination because "technically" you meet the guidelines.
>>
>> In any case, overimposing a precedence on the techniques used to evaluate (not the techniques used to develop) could lead to owners claiming conformance but not conforming to WCAG 2.0, and then pretending that you must use *their* techniques to evaluate, not yours. For example, they could say that they used the keyboard (without a screen reader) to check keyboard accessibility, but if I check with the screen reader running maybe the website is completely unusable. Should I agree with the claim just because they did not test a real world case? Must I consider this website accessible?
>>
>> In conclusion, I am totally against any imposition that gives more precedence to a particular evaluation procedure, of course provided that the evaluation methods follow the WCAG Recommendation.
>>
>> Regards,
>> Ramón.
>>
>>
>> Alistair said:
>>
>>> If I say I have used this set of techniques, and I have undertaken all associated checks, I would not want to be told that I did not conform because I did not use another set of techniques - possibly bound together in some monitoring tool.  Or, think about two EU states using slightly different monitoring tools - which conflict.   Having chaired the Evaluation Methods Task Force for the EuroAccessibility Consortium, and a similar European task force in the initial WABCluster I can say that such conflicts are the rule, rather than the exception. So, a concrete proposal for an addition to step 1.d would be:
>>> "If documented evaluation procedures are provided along with an evaluation statement or WCAG 2.0 conformance claim, the documented evaluation procedures provided should take precedence over any other evaluation procedure when undertaking an evaluation."
>>
>

Received on Tuesday, 20 May 2014 10:53:38 UTC