Usability Testing of the Web Content Accessibility Guidelines – Draft Proposal
David Sloan
Digital Media Access Group, Department of Applied Computing, University of Dundee, Dundee, DD1 4HN, United Kingdom
Tel +44 1382 345598, fax +44 1382 345509 email: dsloan@computing.dundee.ac.uk
Professor Helen Petrie
Director, Sensory Disabilities Research Unit and National Centre for Tactile Diagrams, Psychology Department, University of Hertfordshire, Hatfield AL10 9AB, United Kingdom
Tel +44 1707 284629 fax +44 1707 285059 email: h.l.petrie@herts.ac.uk
Summary
This project involves evaluating the usability of the Web Content Accessibility Guidelines (WCAG), with a number of distinct user groups for which the Guidelines are intended. The usability evaluation will involve Version 2.0, the proposed revision of the Guidelines. Version 1.0, the current version of the guidelines, will also be used as a comparison.
Significant user groups will be identified and used throughout the testing process, including both novice and experienced web designers.
The testing will concentrate on eliciting qualitative data, although some quantitative data will also be and will follow on from the work carried out on evaluation of Version 1.0 of the WCAG by Colwell and Petrie (1999).
The main deliverable will be a report documenting the user groups tested, a description of, and the results of, the testing procedure, and conclusions drawn from the findings. It is hoped that results from the study will produce valuable input into the final content and appearance of Version 2.0 of the WCAG.
Objectives
The main question to be answered is "Are the WCAG usable to the intended audience", bearing mind that the intended audience could be virtually anyone. To answer this question, qualitative and quantitative data, from usability testing with a variety of user groups, will be used to assess current strengths and weaknesses of the two versions of the guidelines.
The findings of the testing stage will be passed back to the Web Accessibility Initiative as part of the development stage of the WCAG version 2.0.
Digital Media Access Group
The Digital Media Access Group (DMAG) is a research group based within the Department of Applied computing at the University of Dundee. The main focus of the group is accessibility and usability of digital resources, primarily web based resources.
Research has included the development of a methodology for auditing the accessibility of web sites, and for generation of clear and prioritised recommendations aimed at maximising the accessibility of the resource. Members of DMAG have presented at:
•	ACM Conference on Assistive Technologies (ASSETS) 2000
•	Conference on Universal Usability (CUU) 2000 in Washington DC, USA
•	CHI 2001, Seattle
•	WWW10 (2001), Hong Kong
DMAG have also carried out accessibility
Current and past clients of the group include:
•	accessibility audits of Scottish Parliament, Bank of Scotland, and a number of web sites in UK Higher Education
•	accessibility advice and consultancy on the Scottish Higher Education Funding Council (SHEFC) funded Communication and Information Technology (C&IT) Web Tools Projects
•	accessibility advice 
•	accessibility support to the ACM
•	collaboration over a report on accessibility issues for the proposed e-University.
Sensory Disabilities Research Unit
The Sensory Disabilities Research Unit (SDRU) at the University of Hertfordshire conducts research on how new technologies can assist in overcoming the problems faced by disabled and elderly people in their day to day lives and on how to design new technologies so that they are usable by the broadest possible range of users. Its work specializes in the elicitation of user needs and requirements, the design of new technologies, the adaptation of existing technologies and the evaluation of systems. The SDRU has a multi-disciplinary team of researchers with backgrounds in psychology, computer science and information science, and considerable expertise in research in this area. A high level of end-user involvement takes place in all work and the SDRU has an enthusiastic and highly critical panel of disabled and elderly people who participate in studies. Good links have also been established with many local and national organizations of and for disabled and elderly people.
The SDRU has undertaken a number of activities related to Web accessibility, including:
•	Conducting a user-based evaluation of a draft version of the WCAG1.0; the results of that evaluation were fed back to the WAI 
•	Advised the British Disability Rights Commission on Web accessibility issues and the development of their Web site
•	Advised the British Museum on the development of their Web site 
•	Taught courses for blind computer users on using the WWW (with both specialist browsers and screen readers used in conjunction with standard browsers) and on Web authoring
Background
The profile of web accessibility has never been higher, with the implications of legal requirements following legislation such as Section 508 of the Rehabilitation Act in the US, and the Disability Discrimination Act in the UK, on those involved in web design. The W3C’s Web Accessibility Initiative has played an invaluable role in raising awareness in accessible design issues, notably through development of the WCAG.
Yet, it is not yet clear how widely understood the important issue of Web accessibility is, in the UK and elsewhere, amongst the people developing Web resources – either the organizations which wish to have Web sites or those who are able to develop the Web sites. Nor is it clear whether Web developers find the Guidelines understandable and easy to use. An initial investigation of the usability of a draft version of the Guidelines by Colwell and Petrie (1999), albeit undertaking with students who were still learning Web development, showed that there were a number of areas of confusion concerning the Guidelines.  This initial conclusion has been reinforced by numerous requests from professional Web developers for clarification of the Guidelines.  Furthermore, previous research on the use of guidelines as a tool for computer software and interface developers (for example De Souza and Bevan, 1990; Mosier and Smith, 1986) has shown that developers had significant difficulties in interpreting and following guidelines.
There is thus a need to ensure that the forthcoming version of the WCAG is designed in such a way as to maximise usability amongst the target audience.
Aims of the Research
Therefore the proposed research will undertake two studies, one on the level of knowledge of the Web accessibility, and the other of the usability of the draft revision of the WCAG guidelines with the following aims:
•	To understand the current level of knowledge of Web accessibility issues and attitudes towards Web accessibility amongst organisations wishing to have Web sites and those who develop the Web sites
•	To understand what kinds of information and tools Web authors would like to see available to them to assist in developing accessibility Web material
•	To measure the usability of the proposed draft of the WCAG guidelines
•	To propose any improvements to the guidelines which can be incorporated immediately
•	To develop an understanding for how the information provided in the guidelines can be made available to Web authors in more usable, effective ways
Study 1: Knowledge of and attitudes towards Web accessibility issues amongst the Web authoring community
Participants
Two target populations will be studied and approximately 15 people from each will be interviewed:
•	Individuals within organizations with responsibility for developing a web site for the organization (who has such a responsibility will vary between organizations) or with authorizing the development of a web site
•	Individuals involved in web site development
Methodology
As these are likely to be busy individuals, a brief interview schedule, lasting no more than 20 minutes will be developed, which can be administered by phone or face-to-face.  The interview schedule will cover:
•	Knowledge of how people with disabilities use the Web
•	Knowledge of the WCAG guidelines for developing accessible Web material
•	Importance to the organization of developing an accessible Web site
•	Types of support needed to produce an accessible Web site
Study 2: Usability of the Web Content Accessibility Guidelines by professional Web authors
Participants
Two target populations will be studied and approximately 12 people from each will be included in the study:
•	“Professional” web authors: those who have a considerable knowledge of HTML and associated web languages and technologies
•	“amateur” web authors: those who have limited knowledge of HTML and associated web languages and technologies, who use a web authoring package to produce their web material
It is thought important to include these two populations, as both types of web authors are very active in producing material for the Web and the WCAG guidelines need to be useful and effective for both populations.
Methodology
The methodology will be similar to that used by Colwell and Petrie (1999) which was found to be effective and yielded much useful information.  
Where possible the study will be conducted in participants’ normal web authoring environment – their office or home. Permission will be sought to video the session for later detailed analysis (where this is not possible, the researcher will take detailed notes of the session).
Before the session itself, participants will be asked to familiarise themselves with the WCAG guidelines and keep a log of the amount of time they spend on that task.
For the session itself, participants will be provided with some content material and asked to author this into an accessible set of Web pages.  The material will be carefully chosen so that it provides a range of challenges in terms of accessibility.
Participants will be asked to provide a “concurrent think aloud protocol”, that is, to provide a (brief) running commentary on what they are doing and what problems they are facing. The researcher will sit with the participant, observing the authoring process and prompting them where necessary to describe their actions. This technique has been shown in many studies to provide rich data of the participants understanding of the situation.
Suggested Deliverables
Project Plan. Including outline of each stage, schedule, methodology, resources, risk management etc.
Background Research – an analysis of prior research into usability of guidelines
Testing Methodology – before evaluations commence, the proposed evaluation procedure will be fully documented
Final Report – including results of the evaluation, analysis of the findings and recommendations based on the findings.
Schedule
To be agreed with the WAI.
It is estimated that the project would take 6 months to complete.
Costing and Resources
An application will be submitted to the Leverhulme Foundation for funding of the project.
Researchers from both the University of Dundee and University of Hertfordshire.
Further details of costing and resources have yet to be finalised.
References
Colwell, C. and Petrie, H. (1999). A preliminary evaluation of the WAI guidelines for producing accessible web pages. In C. Bühler and H. Knops (Eds.), Assistive technology on the threshold of the new millennium. Amsterdam: IOS Press.
De Souza, F, and Bevan, N (1990) The use of guidelines in menu interface design: evaluation of a draft standard. In Diaper, D., Gilmore, D., Cockton, G. and Shackel, B. (eds) Human Computer interaction – INTERACT ’90, pp435-440.
Mosier, J. and Smith, S. (1987) Application of guidelines for designing user interface software. Behaviour and Information Technology, Vol. 5, No. 1 pp39-46