Re: Comments to "Involving Users in Web Projects for Better, Easier Accessibility" and "Involving Users in Evaluating Web Accessibility"

Dear Henrike Gappa, Gabriele Nordbrock, Carlos Velasco,

Thank you again for your comments on WAI’s Involving Users documents, which are archived at http://lists.w3.org/Archives/Public/wai-eo-editors/2010Jan/0000.html and have an initial reply at http://lists.w3.org/Archives/Public/wai-eo-editors/2010Jan/0008.html

As I mentioned previously, for "Involving Users in Web Projects for Better, Easier Accessibility" (http://www.w3.org/WAI/users/involving) I think the main issue is a misunderstanding of the scope. This document/web page talks broadly about including users from the beginning of different types of projects; not about a test methodology, or even specifically about testing at all.

Both documents are intended to be _short introductions_, primarily for people who know little or nothing about involving users. They are intended to be encouraging, and not scare people away from involving users because they think it is too complex or costly. They are not intended to be comprehensive guidance. We agree this is a challenge requiring cautions.

EOWG has reviewed and discussed your comments; replies are below. The documents are now updated at:
- Involving Users in Web Projects for Better, Easier Accessibility http://www.w3.org/WAI/users/involving
- Involving Users in Evaluating Web Accessibility http://www.w3.org/WAI/eval/users 

Kind regards, 
~Shawn, for W3C WAI Education and Outreach Working Group (EOWG)

===

I. Involving Users in Web Projects for Better, Easier Accessibility (http://www.w3.org/WAI/users/involving)

Comment:
We support wholeheartedly encouraging web site developers and owners to conduct user testing for better accessibility. However, we feel that the limitations of the proposed test methodology needs to be addressed much more clearly.

EOWG reply:
This document does not present a test methodology; it is an introductory document about involving users in a wide range of web-related projects, including developing technical specifications and accessibility standards. That it is an introductory document is stated in:
- "This page gets you started...",
- "Below are the basics that you can do yourself to include users in your projects.[edited] If you have the resources, consider getting assistance from accessibility, disability, and user-centered design specialists.", and
- "This document briefly addresses a few points of a very complex topic."
Furthermore this page talks more broadly about including users in the beginning of the development process, and not specifically about testing.

CHANGED:
- Edited the Introduction, including adding "...early and throughout different types of projects. A separate page focuses on including users in evaluation for web development projects."
- Edited: "Below are the basics that you can do yourself to include users in developing websites and other web products." to "Below are the basics that you can do yourself to include users in your projects."
- Under "Including Users in Implementation" moved out of the bullet into its own paragraph: "For more in this, see Involving Users in Evaluating Web Accessibility, especially..."

Comment:
For instance, to stress that the test results obtained with only a small group of disabled users can only be understood as informative, also in regard to the generalization of test results to users with the same characteristics, e.g., blind or screen-reader users. There is a lot of diversity among people with the same disability even when utilizing the same assistive technology.

EOWG Reply:
We agree with this comment and are aware of this issue. We have included several cautions and reminders throughout both documents. In particular, this is covered in:
- "Caution: Carefully consider all input and avoid assuming that input from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the web, nor know enough about other disabilities to provide valid guidance on other accessibility issues. Getting input from a range of users is best." is included in both documents, in strong, highlighted text.
- And the four paragraphs under "Getting a Range of Users" at http://www.w3.org/WAI/users/involving#diverse

Comment:
Furthermore, to our knowledge, it is a precondition to have a deep insight into accessibility guidelines and, at least to some extent, experience with assistive technologies to really employ user testing for better accessibility and to create accessible web applications.

EOWG Reply:
This document is not limited to testing and creating web applications. ("Involving Users in Evaluating Web Accessibility" is directly related to testing/evaluation.) This document is also for people who are new to web accessibility and want to get some general knowledge and experience with how people with disabilities use the web, including high-level project managers, project stakeholders, policy makers, and other situations where deep knowledge of accessibility guidelines is not a prerequisite.

Comment:
In other cases, its benefit is more of motivational and educational nature which is of great value and thus highly recommendable. The described issues are somehow mentioned in the document, but should be pointed out much more explicitly. For instance when contrasted with the section "More Efficient Development", the limitations of the proposed methodology will not be identified most likely by the uninformed user.

EOWG Reply:
We don't understand this comment. Could you be specific about what issues you are referring to, and what further advice you would suggest for this document (keeping in mind our other replies)?

Comment:
Besides this, we are afraid it might not become clear to the reader which user groups are addressed by the note. Most of the time the note names only people with disabilities, sometimes it is people with disabilities and older people and sometimes it is only "users", for instance, in the section "More Efficient Development".

EOWG Reply:
Generally the document uses "users" as a shortcut for "people with disabilities and older people with accessibility needs due to aging". It would be too awkward to use the full phrase each time. The closing sentence of the Introduction establishes the user group(s) addressed: "This page... involving people with disabilities and older people with accessibility needs due to aging, throughout your projects."[The CHANGE below further clarifies this.]

CHANGED: Edited in introduction: "...involving users — specifically people with disabilities and older people with accessibility needs due to aging..." then used just "users" mostly throughout.

Comment:
Suggesting as test methodology to "ask a lot of questions" to gather user data is vague. We propose to be more concrete. Since tests sessions have a time limit, it is advisable for many test purposes to develop at least a questionnaire guide to ensure that all relevant issues are touched and that test results are comparable in case there are several test participants.

EOWG Reply:
Again, please note that this document does not provide a "test methodology". Step 5 says, "For more in this, see Involving Users in Evaluating Web Accessibility...." – acknowledging that there is more to it. This document is to provide some simple guidance for getting users involved in web projects, including projects that do not involve tests at all.

CHANGED: Under "Including Users in Implementation" moved out of the bullet into its own paragraph: "For more in this, see Involving Users in Evaluating Web Accessibility, especially..."

---

II. Involving Users in Evaluating Web Accessibility (http://www.w3.org/WAI/eval/users)

Comment:
Our first comment in regard to this note is -- as it was for "Involving Users in Web Projects for Better, Easier Accessibility" -- that the limitations of the described evaluation methodology need to be pointed out more clearly. We are in favour of user testing, and agree that user testing may reveal accessibility issues that would not have been detected via standard conformance testing. However, it is also important to note that many accessibility errors will definitely not be detected by user testing alone. Therefore, different from what is stated in the introduction of this article, to our knowledge, conformance checks to all relevant accessibility guidelines are not only "important" but need to be understood as the basic of all accessibility evaluations.

EOWG Reply:
We agree that conformance testing is an essential aspect of web accessibility evaluation. There is an entire section in this document that is listed in the Page Contents, "Combine User Evaluation with Standards", which says: "Involving users with disabilities in evaluation has many benefits; however, it alone cannot determine if a website is accessible. Combine user involvement with evaluating conformance to WCAG to ensure that accessibility is provided to users with a range of disabilities and situations." Additionally, at the top of the document, it is placed in context: "This page is part of a multi-page Evaluating Web Accessibility resource suite that outlines different approaches for evaluating web accessibility."

CHANGED: To clarify that these are complimentary approaches, changed in that sentence from "different approaches for" to "different aspects of", resulting in: "This page is part of a multi-page Evaluating Web Accessibility resource suite that outlines different aspects of evaluating web accessibility."

Comment:
We would also suggest conducting a full review instead of a preliminary one and fix all errors, before bringing in users to avoid operation errors of the assistive technologies due to accessibility errors in the code.

EOWG Reply:
There are many cases where it would not be good to do a full conformance evaluation and fix all identified errors before bringing in users. Often it is best to get user input as early as possible, even with early prototypes that are not perfect–-especially when developers and designers are new to accessibility and just learning how people with disabilities use their products.
We agree that accessibility errors in the code can impact user interaction, and have addressed that in this document with the following:
- "The initial review identifies any significant accessibility barriers to fix before evaluating with users."
- "Web accessibility depends on several components of web development and interaction working together, including web browsers, assistive technologies (AT), and web content.... For example, if a user who cannot use a mouse has trouble with keyboard access, it could be because:..."

CHANGED: Edited first paragraph under "Initial Review" to point to Conformance Evaluation as well.

Comment:
In the section on "Range of User Evaluation" informal vs. formal usability evaluation are compared. We think it would be important to also explain here differences of the outcome in terms of significance and validity.

EOWG Reply:
We have tried to keep this document as short as feasible. We assumed that readers will understand that "Informal evaluation of a specific accessibility issue can be as simple as asking someone you know... even your grandmother" will give you only informal outcomes; and that "Formal usability testing of a website follows established protocols to gather quantitative and qualitative data from representative users performing specific tasks." will give you outcomes with greater significance from a research perspective. We don't see that additional explanation is worth increasing the length of the document.

Comment: Also gains and limitations of involving only a few people with disabilities as stated in the section "Basics", should also be pointed out clearly to the reader, so the reader is able to conclude correctly what will be an appropriate test scenario for her purposes.

EOWG Reply:
We agree with this comment and are aware of this issue. We have included several cautions and reminders throughout both documents. In particular, this is covered in:
- "Caution: Carefully consider all input and avoid assuming that input from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the web, nor know enough about other disabilities to provide valid guidance on other accessibility issues. Getting input from a range of users is best." is included in both documents, in strong, highlighted text.
- And the referenced to the four paragraphs under "Getting a Range of Users" at http://www.w3.org/WAI/users/involving#diverse

Comment:
In the section "Drawing Conclusions and Reporting", the editors state the need to be careful when "... drawing conclusions from limited evaluations ....". However we feel that this is not explicit enough, because valid conclusions cannot really be drawn from such user testing, and this should be clear to the reader. The problem is not so much the lack of statistical significance as mentioned, but the limited validity and generalisability of test results.

EOWG Reply:
This document is intended to cover a range of situations, and some studies may be robust enough to provide statistical significance and sufficient validity of test results. We agree that a more explicit caution against generalizing results is warranted in this document.

CHANGED: Added a new sentence (the middle one): "Be careful drawing conclusions from limited evaluations or studies. Results from only a couple of people with disabilities cannot be generalized to apply to all people with similar disabilities or people with other disabilities. See the Caution above."

Comment:
In the section on "Analyzing Accessibility Issues", it is proposed to assign occurring accessibility issues to the origin, e.g., "the developer did not markup/code the web page properly" or "the user's AT isn't handling the markup properly". From our experience with accessibility audits, this can only be achieved by accessibility experts, which means for a reliable judgement, that the evaluator needs to have deep knowledge about HTML, accessible web coding and standard behaviour of AT. Otherwise, such mappings are in danger of being error-prone.

EOWG Reply:
We agree that expertise is often required to accurately determine which components are responsible for identified accessibility problems.

CHANGED: Changed "For any accessibility problems you identify, determine which components are responsible." to "Accessibility problems can be caused by one or more different components."

Comment:
Finally, the section "More Information and Guidance" provides information on user testing specifically for usability professionals. Here, we do not really understand what is meant by "usability testing for accessibility". Both, user testing for usability as well as accessibility issues are based on methods and techniques derived from Psychology and related sciences, e.g., work psychology or software ergonomics. Thus to our understanding there are usability tests and accessibility tests which follow different goals. They might overlap in regard to certain sub-goals and methodologies employed, yet, the goal is quite different and both disciplines should not be intermixed to our understanding.

EOWG Reply:
Without getting into formal definitions, here is a brief explanation of our perspective: The term "usability testing" generally means having real users perform real tasks with a product in order to gather quantitative and/or qualitative data on how well people can use the product. Then "usability testing for accessibility" is having real users with disabilities perform real tasks with a product in order to gather quantitative and/or qualitative data on how well people with disabilities can use the product, and on any accessibility barriers. We believe that usability testing and accessibility evaluation should be more integrated. In any case, the phrasing "usability testing for accessibility" is from a non-W3C/WAI resource. The phrasing we used in the previous section is: "When defining usability tests specifically to find accessibility issues...", and we have edited that to be even more clear, as noted below.

CHANGED:
- Edited Note for usability professionals section to clarify the bullets apply "when specifically exploring accessibility barriers", and added: "Note that is is also important to evaluate general usability, user satisfaction, and other such criteria for users with disabilities."
- Indicated links off the W3C/WAI website with an icon with alt="links off WAI website"

===

We hope that you find the documents satisfactory with these changes.

Regards,
~Shawn, for W3C WAI Education and Outreach Working Group (EOWG)


-----
Shawn Lawton Henry
W3C Web Accessibility Initiative (WAI)
e-mail: shawn@w3.org
phone: +1.617.395.7664
about: http://www.w3.org/People/Shawn/





Shawn Henry wrote:
> Dear Henrike Gappa,
> 
> Thank you again for you recent comments. I have had a chance to review 
> them and have a general reply for now.
> 
> For "Involving Users in Web Projects for Better, Easier Accessibility" 
> (http://www.w3.org/WAI/users/involving) I think the main issue is a 
> misunderstanding of the scope. This document/web page talks broadly 
> about including users from the beginning of development processes; not 
> about a test methodology, or even specifically about testing at all.
> 
> Both documents are intended to be short introductions, primarily for 
> people who know little or nothing about involving users. They are 
> intended to be encouraging, and not scare people away from involving 
> users because they think it is too complex or costly. They are not 
> intended to be comprehensive guidance. We agree this is a challenge 
> requiring cautions.
> 
> EOWG will review your comments in detail as soon as we can fit it into 
> our work schedule. We will provide more specific replies then.
> 
> Kind regards,
> ~Shawn
> 
> 
> -----
> Shawn Lawton Henry
> W3C Web Accessibility Initiative (WAI)
> e-mail: shawn@w3.org
> phone: +1.617.395.7664
> about: http://www.w3.org/People/Shawn/
> 
> 
> Henrike Gappa wrote:
>> Dear editors,
>>
>> Please find in the following some comments in regard to the posted notes:
>>
>> - Involving Users in Web Projects for Better, Easier Accessibility 
>> (http://www.w3.org/WAI/users/involving)
>> - Involving Users in Evaluating Web Accessibility 
>> (http://www.w3.org/WAI/eval/users)
>>
>>
>> To "Involving Users in Web Projects for Better, Easier Accessibility"
>>
>> We support wholeheartedly encouraging web site developers and owners to
>> conduct user testing for better accessibility. However, we feel that the
>> limitations of the proposed test methodology needs to be addressed much
>> more clearly. For instance, to stress that the test results obtained
>> with only a small group of disabled users can only be understood as
>> informative, also in regard to the generalization of test results to
>> users with the same characteristics, e.g., blind or screen-reader users.
>> There is a lot of diversity among people with the same disability even
>> when utilizing the same assistive technology. Furthermore, to our
>> knowledge, it is a precondition to have a deep insight into
>> accessibility guidelines and, at least to some extent, experience with
>> assistive technologies to really employ user testing for better
>> accessibility and to create accessible web applications. In other cases,
>> its benefit is more of motivational and educational nature which is of
>> great value and thus highly recommendable. The described issues are
>> somehow mentioned in the document, but should be pointed out much more
>> explicitly. For instance when contrasted with the section "More
>> Efficient Development", the limitations of the proposed methodology will
>> not be identified most likely by the uninformed user.
>>
>> Besides this, we are afraid it might not become clear to the reader
>> which user groups are addressed by the note. Most of the time the note
>> names only people with disabilities, sometimes it is people with
>> disabilities and older people and sometimes it is only "users", for
>> instance, in the section "More Efficient Development".
>>
>> Suggesting as test methodology to "ask a lot of questions" to gather
>> user data is vague. We propose to be more concrete. Since tests sessions
>> have a time limit, it is advisable for many test purposes to develop at
>> least a questionnaire guide to ensure that all relevant issues are
>> touched and that test results are comparable in case there are several
>> test participants.
>>
>>
>>
>> To "Involving Users in Evaluating Web Accessibility"
>>
>> Our first comment in regard to this note is -- as it was for "Involving
>> Users in Web Projects for Better, Easier Accessibility" -- that the
>> limitations of the described evaluation methodology need to be pointed
>> out more clearly. We are in favour of user testing, and agree that user
>> testing may reveal accessibility issues that would not have been
>> detected via standard conformance testing. However, it is also important
>> to note that many accessibility errors will definitely not be detected
>> by user testing alone. Therefore, different from what is stated in the
>> introduction of this article, to our knowledge, conformance checks to
>> all relevant accessibility guidelines are not only "important" but need
>> to be understood as the basic of all accessibility evaluations. We would
>> also suggest conducting a full review instead of a preliminary one and
>> fix all errors, before bringing in users to avoid operation errors of
>> the assistive technologies due to accessibility errors in the code.
>>
>> In the section on "Range of User Evaluation" informal vs. formal
>> usability evaluation are compared. We think it would be important to
>> also explain here differences of the outcome in terms of significance
>> and validity. Also gains and limitations of involving only a few people
>> with disabilities as stated in the section "Basics", should also be
>> pointed out clearly to the reader, so the reader is able to conclude
>> correctly what will be an appropriate test scenario for her purposes. In
>> the section "Drawing Conclusions and Reporting", the editors state the
>> need to be careful when "... drawing conclusions from limited evaluations
>> ....". However we feel that this is not explicit enough, because valid
>> conclusions cannot really be drawn from such user testing, and this
>> should be clear to the reader.  The problem is not so much the lack of
>> statistical significance as mentioned, but the limited validity and
>> generalisability of test results.
>>
>> In the section on "Analyzing Accessibility Issues", it is proposed to
>> assign occurring accessibility issues to the origin, e.g., "the
>> developer did not markup/code the web page properly" or "the user's AT
>> isn't handling the markup properly". From our experience with
>> accessibility audits, this can only be achieved by accessibility
>> experts, which means for a reliable judgement, that the evaluator needs
>> to have deep knowledge about HTML, accessible web coding and standard
>> behaviour of AT. Otherwise, such mappings are in danger of being
>> error-prone.
>>
>> Finally, the section "More Information and Guidance" provides
>> information on user testing specifically for usability professionals.
>> Here, we do not really understand what is meant by "usability testing
>> for accessibility". Both, user testing for usability as well as
>> accessibility issues are based on methods and techniques derived from
>> Psychology and related sciences, e.g., work psychology or software
>> ergonomics. Thus to our understanding there are usability tests and
>> accessibility tests which follow different goals. They might overlap in
>> regard to certain sub-goals and methodologies employed, yet, the goal is
>> quite different and both disciplines should not be intermixed to our
>> understanding.
>>
>> Kind regards,
>> Henrike Gappa, Gabriele Nordbrock and Carlos Velasco
>>
>>
>

Received on Wednesday, 7 April 2010 13:37:24 UTC