WCAG Usability Testing - update and call for comments

Hi there,

Firstly, my apologies for the recent silence regarding our WCAG Usability
testing proposal since I talked at the WG Face-to-Face in Amsterdam.

Helen Petrie and I have submitted a proposal to a UK charitable foundation
in application for funding for the project, and it will take a few weeks to
discover the progress of the application. 

The latest version of the proposal is now on the web at:
http://www.dmag.org.uk/misc/wcag_usability_test.asp 

One change is that we are now looking to run the project over 12 months,
rather than the 6 months I suggested at the face-to-face.

However we are still very keen to ensure that what we do meets the needs and
wishes of the Working group as far as possible, and to that end the proposal
methodology as documented in the above web page is still very much under
development.

As discussed at the face to face, a number of issues came up, with questions
which may require debate:

1. Ecological validity of testing environment:
- many users will not refer to printed or web based guidelines, but will use
them if delivered via a piece of software/a GUI (e.g. as part of an
authoring tool).
- Conversely, it could be argued that the initial research should
concentrate on the web based WCAG as is.
- Some users will be familiar with Bobby, and/or the WAVE, A-Prompt etc,
perhaps as their token accessibility checker. There is a list of these on
the WAI site, but will this list be clearly linked from the WCAG? I would
suggest that it should, if the WCAG is to be optimally usable.

Question: In what format should the guidelines be offered to participants in
the research? Should participants be given access to, or told about checking
tools such as the above? 

2. Subject Characteristics
- The proposal suggests a evaluation with two groups - 12 experienced web
authors and 12 novice web authors
- However, as we know, around the world, web authors are found in many
different environments, with many different backgrounds/levels of
learning/paths to gaining knowledge.
- within the professional web authoring sector there are people who put
increasingly complex information on the web using increasingly complex
solutions, and there are people who use the web to take visual design of
media to new levels. In some cases it may be hard to distinguish one from
the other!
- there are a wide variety of authoring tools out there; there are also
people who code by hand, and those who simply choose the "Convert to HTML"
option in word processing packages.
- yet there was also concern at the face-to-face that the study should not
be "watered down" or compromised in order to be all inclusive.

Question: Should initial testing take place of, as far as possible, a
homogeneous group of users, followed by further (parallel?) studies of other
groups? For example, initially only UK based Dreamweaver users working for
web design companies. 

3. The research will look at how participants use the WCAG to spot
accessibility problems in existing sites, and also when creating sites of
their own, given a carefully chosen design brief. Not only must experts be
on hand to assess sites for accessibility problems, but they must interact
with participants on completion of the task to try to explain the presence
of any accessibility problems, even after participants having had access to
the WCAG. Things we feel important to discover include
- areas where participants had trouble implementing any of the guidelines?
- explanations of why accessibility barriers noticed by experts are present
(including whether participants were aware of this)
- level of satisfaction of participants in what they're created, from a
design point of view, plus any constraints under which they felt the WCAG
placed them.

Question: What further information would group members like to see the
research uncover?

4. Are there any other comments or questions on the methodology set out in
the proposal?

We look forward to hearing your feedback! 

Cheers,
David

Received on Tuesday, 14 August 2001 07:01:28 UTC