RE: Web Accessibility Evaluation and Testing

Ian wrote: "But conformance alone is not enough to guarantee usability."

Generally I agree with this statement, but it would be useful if Ian and 
others would provide some examples of a page or set of pages that 
illustrate the usability issues of where there is agreement on conformance 
to the WCAG 2.0 level A and/or even double AA success criteria, but still 
not considered usable.  Some examples could help us all understand where 
best to address the usability issues. 

My experience has been that many of the quote usability unquote issues 
have to do with the following categories where the page is fully compliant 
to WCAG 2.0 AA:
a. but equally unusable to everyone (e.g. design or task flow issues),
b. but differing usability experiences depending on the browser (user 
agent) and/or assistive technology the person is using,
c. but differing usability experiences depending on the person's 
configuration and/or habits of how they use their browser and assistive 
technology. 

In other words category A is where when there is the same site - but 
everyone is dissatisfied, confused or lost. 
B is where the there is the same site and same person (similar end user 
level), but the issues are really the cause of using a different browser 
and/or level of AT. 
And C is where the site is the same, browsers and AT are the same, but the 
issues are really the cause of the persons having different levels of 
training or knowledge of how to best use that browser and/or AT. 

Other classic "usability" issues or categories of issues are often grouped 
as 
        ease of learning - first time users vs repeat users - and 
everything else is equal.
        translations or availability in my language choice (e.g., Canadian 
French vs Creole French)
 
An example of an equally unusable site is my automobile toll tag 
management web site that complicates the management of account numbers, 
automobile license numbers, tag transponder numbers - all of those with 
the actual toll booth transaction amounts and dates and financial accounts 
(credit card and/or bank account numbers).  The flexibility of having more 
than one automobile in an account and more than one payment method has 
complicated the heck out of managing the stuff.

An example of B is where the same user is using the same web site, but has 
a different user experience when using a different browsers and/or a 
different versions of an assistive technology (e.g. different level of 
JAWS, different level of ZoomText, etc.). 

An example of C is where the same site is used with the same browser and 
same version of assistive technology, but the one users is unfamiliar with 
some of the newer browsing techniques that another user may be familiar 
with such as using assistive technology to navigate by heading vs 
navigating by landmark or navigating a list of links on the page.

Again, some example could help us all understand if the best place to 
address the issue is with WCAG itself, the web site design, the tools 
being used, the users familiarity with the tools, or something else.

I also think that any evaluation and testing methodology needs to consider 
if it is 
        - a new site design verses simply updating content in an existing 
site design
        - a web site verses a web application
        - and design evaluations verses conformance testing verses 
compatibility testing with versions of browsers and AT
 
Regards,
Phill Jenkins, 
IBM Research - Human Ability & Accessibility Center