Re: Comments to "Involving Users in Web Projects for Better, Easier Accessibility" and "Involving Users in Evaluating Web Accessibility"

Dear Henrike Gappa,

Thank you again for you recent comments. I have had a chance to review them and have a general reply for now.

For "Involving Users in Web Projects for Better, Easier Accessibility" (http://www.w3.org/WAI/users/involving) I think the main issue is a misunderstanding of the scope. This document/web page talks broadly about including users from the beginning of development processes; not about a test methodology, or even specifically about testing at all.

Both documents are intended to be short introductions, primarily for people who know little or nothing about involving users. They are intended to be encouraging, and not scare people away from involving users because they think it is too complex or costly. They are not intended to be comprehensive guidance. We agree this is a challenge requiring cautions.

EOWG will review your comments in detail as soon as we can fit it into our work schedule. We will provide more specific replies then.

Kind regards,
~Shawn


-----
Shawn Lawton Henry
W3C Web Accessibility Initiative (WAI)
e-mail: shawn@w3.org
phone: +1.617.395.7664
about: http://www.w3.org/People/Shawn/


Henrike Gappa wrote:
> Dear editors,
> 
> Please find in the following some comments in regard to the posted notes:
> 
> - Involving Users in Web Projects for Better, Easier Accessibility 
> (http://www.w3.org/WAI/users/involving)
> - Involving Users in Evaluating Web Accessibility 
> (http://www.w3.org/WAI/eval/users)
> 
> 
> To "Involving Users in Web Projects for Better, Easier Accessibility"
> 
> We support wholeheartedly encouraging web site developers and owners to
> conduct user testing for better accessibility. However, we feel that the
> limitations of the proposed test methodology needs to be addressed much
> more clearly. For instance, to stress that the test results obtained
> with only a small group of disabled users can only be understood as
> informative, also in regard to the generalization of test results to
> users with the same characteristics, e.g., blind or screen-reader users.
> There is a lot of diversity among people with the same disability even
> when utilizing the same assistive technology. Furthermore, to our
> knowledge, it is a precondition to have a deep insight into
> accessibility guidelines and, at least to some extent, experience with
> assistive technologies to really employ user testing for better
> accessibility and to create accessible web applications. In other cases,
> its benefit is more of motivational and educational nature which is of
> great value and thus highly recommendable. The described issues are
> somehow mentioned in the document, but should be pointed out much more
> explicitly. For instance when contrasted with the section "More
> Efficient Development", the limitations of the proposed methodology will
> not be identified most likely by the uninformed user.
> 
> Besides this, we are afraid it might not become clear to the reader
> which user groups are addressed by the note. Most of the time the note
> names only people with disabilities, sometimes it is people with
> disabilities and older people and sometimes it is only "users", for
> instance, in the section "More Efficient Development".
> 
> Suggesting as test methodology to "ask a lot of questions" to gather
> user data is vague. We propose to be more concrete. Since tests sessions
> have a time limit, it is advisable for many test purposes to develop at
> least a questionnaire guide to ensure that all relevant issues are
> touched and that test results are comparable in case there are several
> test participants.
> 
> 
> 
> To "Involving Users in Evaluating Web Accessibility"
> 
> Our first comment in regard to this note is -- as it was for "Involving
> Users in Web Projects for Better, Easier Accessibility" -- that the
> limitations of the described evaluation methodology need to be pointed
> out more clearly. We are in favour of user testing, and agree that user
> testing may reveal accessibility issues that would not have been
> detected via standard conformance testing. However, it is also important
> to note that many accessibility errors will definitely not be detected
> by user testing alone. Therefore, different from what is stated in the
> introduction of this article, to our knowledge, conformance checks to
> all relevant accessibility guidelines are not only "important" but need
> to be understood as the basic of all accessibility evaluations. We would
> also suggest conducting a full review instead of a preliminary one and
> fix all errors, before bringing in users to avoid operation errors of
> the assistive technologies due to accessibility errors in the code.
> 
> In the section on "Range of User Evaluation" informal vs. formal
> usability evaluation are compared. We think it would be important to
> also explain here differences of the outcome in terms of significance
> and validity. Also gains and limitations of involving only a few people
> with disabilities as stated in the section "Basics", should also be
> pointed out clearly to the reader, so the reader is able to conclude
> correctly what will be an appropriate test scenario for her purposes. In
> the section "Drawing Conclusions and Reporting", the editors state the
> need to be careful when "... drawing conclusions from limited evaluations
> ....". However we feel that this is not explicit enough, because valid
> conclusions cannot really be drawn from such user testing, and this
> should be clear to the reader.  The problem is not so much the lack of
> statistical significance as mentioned, but the limited validity and
> generalisability of test results.
> 
> In the section on "Analyzing Accessibility Issues", it is proposed to
> assign occurring accessibility issues to the origin, e.g., "the
> developer did not markup/code the web page properly" or "the user's AT
> isn't handling the markup properly". From our experience with
> accessibility audits, this can only be achieved by accessibility
> experts, which means for a reliable judgement, that the evaluator needs
> to have deep knowledge about HTML, accessible web coding and standard
> behaviour of AT. Otherwise, such mappings are in danger of being
> error-prone.
> 
> Finally, the section "More Information and Guidance" provides
> information on user testing specifically for usability professionals.
> Here, we do not really understand what is meant by "usability testing
> for accessibility". Both, user testing for usability as well as
> accessibility issues are based on methods and techniques derived from
> Psychology and related sciences, e.g., work psychology or software
> ergonomics. Thus to our understanding there are usability tests and
> accessibility tests which follow different goals. They might overlap in
> regard to certain sub-goals and methodologies employed, yet, the goal is
> quite different and both disciplines should not be intermixed to our
> understanding.
> 
> Kind regards,
> Henrike Gappa, Gabriele Nordbrock and Carlos Velasco
> 
>

Received on Tuesday, 12 January 2010 03:57:19 UTC