WCAG-EM comments. Section 3.5 Report the Evaluation Findings

Dear Eval TF,

Please find below comments for Section 3.5 Step 5 about reporting the 
results.


3.5 Step 5: Report the Evaluation Findings

[Ed] "Reporting your findings is a key element..."

I would change it to bo impersonal, as the rest of the document: 
"reporting the evaluation findings is a key element..." or just 
"reporting is a key element...".


3.5.1 Step 5.a: Provide Documentation for Each Step

Many of the "identified..." components will overlap with the 
"representative sample", it sounds a bit redundant. In particular, 
WCAG-EM now states that "all common web pages" must be part of the 
representative sample.

Again, I would not assume that the "goal" is equivalent to the "level of 
detail" in the report. A "Basic Evaluation" could lead to a more or less 
detailed report of several key barriers, while an "In-Depth Analysis" 
could be summarized in a simplified report for a project manager (or for 
machine processing, without providing advice or suggestions). Technosite 
has a variety of different reports aimed at different audiences (project 
managers and executives, sales staff, content managers, developers...). 
The detail in the report usually depends more on the target audience 
than on the depth of the evaluation.

In particular, for "negative" analysis (that is, quick analysis to 
detect common failures and obtani a binary success/failure result) there 
is no need to evaluate each individual SC nor to document every failure. 
Current wording of "Basic Report" seems to imply that the evaluator must 
carry out a complete, detailed analysis of each web page to identify all 
failures, even if the goal is just to say "good / bad".

Maybe some guidance about possible types of reports could be given, or 
simply mention some possibilities and supplement them with the examples 
in the Appendix.

In addition, I think that the "suggestions for improving" part should be 
an optional component of the report, even for the "in-depth analysis", 
since this goes beyond the evaluation process to the advice/consultancy 
tasks.


3.5.2 Step 5.b: Provide an Accessibility Evaluation Statement (optional)

[WCAG-EM] "WCAG 2.0 conformance claims can only be made for the web 
pages that have been actually evaluated and identified to conform with 
its conformance requirements. Accessibility evaluation statements for 
entire websites can be made according to this methodology when:"

Does this mean that no conformance claim can be made for a complete 
website unless ALL pages are evaluated? This seems too strict, and would 
imply that no logo can be used except for the pages in the sample:

[WCAG 2.0] Note: If a conformance logo is used, it would constitute a 
claim and must be accompanied by the required components of a 
conformance claim listed above.


After the "required components" of the conformance claim, I would also 
add the note about additional SC in other levels to encourage composed 
targets to go beyond a conformance level.

Component "6. Website areas that do not conform". Maybe there are no 
defined areas, but a set of documents that are distributed across the 
whole website (for example, PDF documents prior to the date of the claim).

Examples of "partial conformance statement" would be good. We are 
including a partial conformance statement due to language for PDF 
documents when the context of use includes MacOS or Linux, since PDF is 
not accessibility supported (in any language) on these platforms.

In addition, WCAG-EM includes a mandatory step for the "context of use", 
so I would include at least the optional component about it in the 
Accessibility Evaluation Statement.


3.5.3 Step 5.c: Provide a Performance Score (optional)

We have some concerns about this. Scoring would require a complete 
methodology to achieve a standardised result that all evaluators could 
apply exactly in the same way.

Since the calculations rely on the selected sample, the score may vary a 
lot depending on the final size of the sample. For example, a single 
failure in a single web page will imply three different scores if the 
sample size is 15, 30 or 60 pages.

In addition, the term "applicable Success Criteria" (even with the "as 
per 3.1.3 Step 1.c" addition) may be confusing or considered differently 
from evaluator to evaluator. For example, some evaluators could consider 
that "SC 2.3.1 Three flashes or Below" only applies if there is flashing 
content (failing or not), while others could consider it always applies. 
The score result would then vary even if the results are the same. The 
note is even more confusing, since it uses "apply" with the other 
meaning: "there is content for which the SC can be evaluated".

Note also that the "Per Instance" approach

Lastly, as a side note I must say that our experience with overall 
scores is not always good. Some clients tend to interpret results in two 
ways:

- Low score (<30%): "Our website is so bad that we will need a lof of 
effort to redesign it. We cannot afford so many changes, so we will 
better do nothing unless we are forced to it (law requirements, sues / 
complains, etc.)"

- High score (>70%): "Our website is good enough as it is, we don't want 
to waste extra time and resources improving it even more. We will solve 
issues only if someone complains or when our annual profits are better".

Kind regards,
Ramón.

---
Ramón Corominas
Accessibility specialist
Technosite - Fundación ONCE
+34 91 121 0330

Received on Monday, 22 October 2012 13:33:22 UTC