RE: Evaluation Resource Suite

Alistair,

Thanks much for sharing your comments. Indeed we did discuss these very
points last week; although the discussion may not be easy to discern
from the minutes.

We discussed more clearly distinguishing three levels, where a
conformance or technical evaluation to the guidelines does not require
usability testing and a comprehensive evaluation does include usability
testing. We also discussed related points including that usability
testing should be incorporated throughout a design and evaluation
process; that the guidelines provide the basis for technical
accessibility; that usability testing is also important for "usable
accessibility"; that there are myths and misunderstandings about the
range of disabilities; etc.

I also share the concern that usability testing with participants with
disabilities is a complex issue and it will be challenging to figure out
how to appropriately cover it in this WAI resource.

I am confident that we will figure out an effective way to communicate
these important points through our process of editing, reviewing,
discussing, and revising within EOWG and with others.

Best regards,

~ Shawn


> -----Original Message-----
> From: w3c-wai-eo-request@w3.org 
> [mailto:w3c-wai-eo-request@w3.org] On Behalf Of Alistair Garrison
> Sent: Friday, October 15, 2004 10:22 AM
> To: EOWG; Judy Brewer
> Subject: Evaluation Resource Suite
> 
> 
> 
> Dear EOWG, 
> 
>  
> 
> I have read with interest the minutes from your 
> teleconference dated (minutes - 
> http://www.w3.org/WAI/EO/2004/1008.html), and was reminded of 
> the discussion we had about this topic in our Dublin 
> Face-To-Face last year (minutes - 
> http://www.w3.org/WAI/EO/2003/0905.html). 
> 
>  
> 
> Now, as then, I have certain concerns about the linking of 
> Conformance testing WCAG with Usability Testing (relating 
> only to people with
> disabilities).    I felt I should write something when I read 
> an interview
> taken from "Accessibility study of bbc.co.uk: Problems faced 
> by users with disabilities" [PDF
> <http://www.bbc.co.uk/commissioning/newmedia/pdf/BBCi_Accessib
> ility_Study_7-
> 10-02.pdf>  file, 1.7 mb; DOC
> <http://www.bbc.co.uk/commissioning/newmedia/pdf/BBCi_Accessib
> ility_Study_7-
> 10-02.doc>  file, 2.2 mb]).  The specific extract is as follows.
> 
>  
> 
> S: Do you use guidelines such as the WAI, and if so how easy 
> and practical are they to adhere to?
> 
>  
> 
> U: We have automated scripts that validate our code before it 
> is published. But the guidelines fall a long way short from 
> being a recipe for an automatically accessible site. They 
> stop you from making glaring mistakes, but the majority of 
> the work is in designing good navigation.
> 
>  
> 
> S: Have you developed your own in house guidelines?
> 
>  
> 
> U: Not formally.
> 
>  
> 
> S: Do you test with users?
> 
>  
> 
> U: Always.  In addition, we employ a consultant (who uses a 
> screen reader) to review the sites that we build.  Having a 
> visually impaired user review the site makes more difference 
> than any amount of guideline following.  He sends us audio 
> tapes of the screen reader output, and I play these to the developers!
> 
>  
> 
> I am 110% behind the statement that usability testing is the 
> best way to provide a truly useable website for your users, 
> however, the point must be made that to undertake a proper 
> usability test requires a great deal of skill, time and 
> effort.  This is especially true when it comes to selecting 
> users that are truly representative of your target audience 
> (where attendance by people with disabilities in the target 
> audience should be strongly encouraged).  
> 
>  
> 
> The mistake which must be avoided is clearly shown in the 
> example above, by taking advice from this single 'disabled' 
> user (whose 'usability matrix' is clearly unknown i.e. all 
> those things that add or detract from their user experience 
> e.g. assistive technology, assistive technology set-up, 
> computer system, connection speed, level of experience, 
> disability, etc.) this company runs the risk of making 
> changes to their website (used by 1000s) that could 
> dramatically improve the user experience for some, but leave 
> others potentially worse off.
> 
>  
> 
> It is clear from reading such articles, that people have 
> forgotten (or never
> understood) the fact that the Guidelines are a collection of 
> improvements (from pan-disability organisations world-wide) 
> whose implementation will in all probability (and depending 
> on the level of Conformance) aid a large range of users with 
> disabilities.  
> 
>  
> 
> In light of this, I would encourage a more supportive message 
> to be sent out regarding the benefits of Conformance testing 
> 'Technical Accessibility', with a separate statement 
> suggesting that when usability testing is done for a website 
> all efforts should be made to include users from the target 
> audience who have disabilities. 
> 
>  
> 
> Very best regards 
> 
>  
> 
> Alistair 
> 
>  
> 
> Alistair Garrison, Managing Director
> 
> Accessinmind Limited UK Filial
> 
> +46 (8)44 65 287
> 
>  
> 
> IMPORTANT: This e-mail, including any attachments, is for the 
> addressee   
> 
> only.  It may contain privileged and/or confidential 
> information.  If it   
> 
> has come to you in error, please notify the sender 
> immediately.  If you   
> 
> are not the intended recipient you must not use, disclose, 
> copy, print,   
> 
> distribute or rely on its contents.  All e-mails and any 
> attachments are
> 
> believed, but not warranted, to be virus free.  However, all 
> e-mails   
> 
> should be virus checked before being downloaded and we accept no  
> 
> responsibility therefore.
> 
>  
> 
> 
> 
> 
>

Received on Friday, 15 October 2004 15:57:10 UTC