Comments from BrailleNet and experts on 5.C "Provide a Performance Score (Optional)"

Hello,
Please find below our comments on section 5.C: Provide a Performance 
Score (Optional).
"[Review Note: Feedback on this section is particularly welcome. Another 
scoring approach being considered is to instead record failures of 
success criteria. Also being considered is tracking those that are not 
applicable. Are there other simple yet effective scoring approaches? Is 
there an approach preferred by the reviewer? For example, how do these 
scoring approaches work in practices? Should the scoring be based on 
applicable Success Criteria only? Is scoring be a good idea at all?]"
Before answering the question on scoring, we would like to emphasise 
That all criteria don't have the same impact on accessibility. This 
methodology should recall that scoring indicates criteria that have been 
taken into account but only shows the work that still needs to be done. 
It cannot be in any case a metric for the real accessibility level of 
the site. On a web site, it may happen that 98% of the criteria are met, 
but if the site goal is to create an account via a form containing a 
CAPTCHA without alternative, the site will remain 100% inaccessible for 
some people. On the other hand, a web site may have many invalidated 
success criteria (because many decorative images have not been labelled 
properly, because they have been indicated by many contributors on many 
parts of the site), and it can get a relatively low scoring although it 
is more accessible than the site that had 98% accessibility, mentioned 
above.
Moreover, the same criteria can be more or less important for 
accessibility, according to where it is applied. If a keyboard trap is 
located at the top of a page, the user will not be able to enter the 
site that is 100% inaccessible. If the keyboard trap is located at the 
bottom of the page, it may be disturbing to refresh the page in order to 
go back to the top, but the user is still able to access the content of 
the site.

Statistics do not indicate an accessibility level, they just show how 
accessibility criteria have been taken into account in a defined period. 
In order to better measure the accessibility level of a site, it is 
necessary to add to scoring a list of barriers prioritized according to 
their relative importance on barriers in information access that are 
identified by the user of Assistive Technologies based on use cases and 
user scenarios. These use cases or scenarios are based on core 
functionalities of the site/application.

Scoring is a necessity, in particular, for organizations who decide to 
progressively implement Web accessibility. They usually already use one 
anyway, and it is a good thing that the methodology deals with scoring. 
But it is also the W3C-WAI responsibility to clearly explain the limits 
of a score only based on the amount of validated success criteria to 
evaluate the real accessibility level. Because the first step of WCAG (A 
level) is so difficult for some websites, the three levels A, AA, AAA 
are not sufficient to fall within a progressive enhancement approach of 
a Web accessibility level. The relevant metrics to measure the 
accessibility level of a Web page, include the criteria which really 
represent a barrier for some users in specific contexts (like CAPTCHA, 
keyboard trap, etc.). This metrics is still to be invented.
In 5.c, the text should say more clearly that scoring helps to identify 
indicators of the accessibility levels of a web site, but even if the 
scoring of failed success criteria is low, the accessibility barriers 
may be so important that some users will not be able to get to the 
information provided by the web site or to use the web site's essential 
functionalities.

The experts of our group who use scoring regularly find that the three 
scoring approaches provided by the methodology are really useful and 
complementary. For that reason, the methodology should recommend to use 
all three of them in order to get a better overview of the scoring 
performances.

As far as the second (per Web page) and the third (per instance) scoring 
approaches are concerned, we find that the introductory sentences do not 
reflect or clearly explain what each of them really calculates. The 
second approach, first sentence, says that "this score calculates an 
average ratio over each web page".

Many of us use another scoring approach to calculate a scoring per Web 
page: they calculate for each page the average of success criteria that 
have been met (passed / (passed+failed)) and do the average through the 
whole site. For example, if one page has 50% passed, and another has 
100% passed, we calculate as follows: (50+100)/100*2 (that is to say the 
sum of the percentage of passed criteria for each page / number of 
evaluated pages*100). So 75% in our example. With the formula described 
in WCAG-EM, the score would be different if the number of applicable 
criteria (passed+failed) is different from one page to the other.

We wonder why the formula we have described above has not been chosen 
and would appreciate explanations on the choice of the formula in 
approach 2. The same questions are raised for approach 3.

We think that the formula should be illustrated by precise mathematical 
formulas and real examples. We have constructed several examples on a 
fictive web site and would be happy to send them to the group as soon as 
they have been translated into English.
We found this discussion on scoring really interesting and are happy to 
share more on that topic if necessary.
Best regards
-- 
Sylvie Duchateau
for Association BrailleNet and group of accessibility experts

Received on Tuesday, 16 April 2013 13:55:30 UTC