3 comments on Conformance Evaluation Methodology

Comments:

1. Abstract
A little more information is needed on what else is NOT covered or 
applicable with this methodology.  It is clear what this methodolgy: "only 
addresses accessibility evaluation of websites after their development . . 
. "
but the last line in the paragraph should be expanded with more examples 
of what this methodolgy is NOT good for, without having to go to the 
reference.  For example, the Abstract does mention "maintaining 
accessibility", while it could be expanded to also reference the following 
sections (with or without the links - cause they may change):
        Ongoing Monitoring      
http://www.w3.org/WAI/eval/considerations.html#ongoing
        Dynamically generated web pages  
http://www.w3.org/WAI/eval/considerations.html#dynamic
but one are that is not cover with the methodology and not in the 
additional resources is 
        doing an 'accessibility evaluation during the design phase' (not 
just during development) but actually evaluating wireframes, visual 
mock-ups, information architecture, and design choices where there may or 
may not be any code running in a browser yet. Another area NOT covered is 
doing an 'accessibility evaluation on implementation choices', where 
various UI tool kits (e.g. jQuery, Dojo, etc.), custom widgets, template 
layouts, and basic HTML choices are evaluated BEFORE implementation 
begins.  This is often a very useful step when doing a 're-design' of an 
existing website.

2. 5 step process seems overly complex.
Could it be simplified into a 3 step process, but still cover all the 
activities? 
Steps 1, 2, and 3 really could be combined into a single "Step 1. 
Establish evaluation requirements and scope", but still include most of 
the activities and tasks. This methodology is mostly written for an 
external view, meaning from someone who is NOT the site owner, but a third 
party providing an evaluation service that has no knowledge of the site or 
development team before hand.  So some of these activities and tasks will 
seem "obvious" to some, while to others it will seem "I have no idea, I'm 
not the site developer".  Step 4 Audit. . .  should remain as a single 
step, and Step 5 Reporting . . . should remain a single step.

3. Step 4 Audit
First, change the title to "Evaluate the selected sample" or "Diagnostic 
Assessment of selected sample".  The term 'audit' has a different 
conotation.  Everyelse in the document the term "evaluation' is used, as 
in Step 5. Reporting the evaluation results - NOT reporting the audit 
results. 
Then, this section needs some more explanation of actually how to do the 
evaluation.  I recommend 3-4 methods: 1. using automated tools, 2. using 
assistive technology, 3. using manual inspections by subject matter 
experts, 4 (optional depending on scope and phase in development) using 
usability test participants who have disabilities.  This is the "meat" of 
the methodology and appears to me to be totally missing from the document. 
Each WCAG 2.0 Success Criteria can be evaluated in many of the ways, but 
there are "most effective ways", and that should be explained. I recommend 
setting up a framework to help guide evaluators on which combinations of 
methods (tools, manual, AT, and user test) are best for each particular 
success criteria - not in a perscriptive way, but in an explanatory and 
educational way. Just as you wouldn't look up each word in a dictionary to 
do a spell check, you would use an automated spell checker, but you would 
also commission an editorial review too, correct? Examples like the 
following should be provided in Step 4: a.  automated checking tools find 
the existance of videos, but not too good yet at telling if there are 
captions provided; b. automated checking tools find the existance on 
labels on form controls, but not if they are visually associated to the 
most approriate control.  c. automated checking tools check for the 
existance of alt text on images, but not if the text alternative is most 
appropriate for the particular important images such as links, d. etc. 

____________________________________________
Regards,
Phill Jenkins, 
Senior Engineer & Business Development Executive
IBM Research - Human Ability & Accessibility Center
http://www.ibm.com/able
http://www.facebook.com/IBMAccessibility
http://twitter.com/IBMAccess