Re: Ethics of user testing and people with cognitive disabilities

Hi David -

The Research Questions Task Force has formed and is in its recruiting phase:

https://www.w3.org/WAI/APA/task-forces/research-questions/

Announcements have gone out in various places, including a repeat to the 
WAI Interest Group yesterday:

https://lists.w3.org/Archives/Public/w3c-wai-ig/2016JulSep/0039.html

Jason White, the initial facilitator, has also reached out to former 
RDWG participants he knew about.

I'm sure the TF would welcome your participation. You can follow 
instructions in the above resources.

I let Jason know yesterday about this thread. It might be a thread we 
should move to the RQTF, where we hope to have a collection of people 
who can offer guidance on ethics questions.

Michael


On 12/07/2016 5:02 AM, David Sloan wrote:
> Hi Lisa, all
>
> The idea of some formal guidance on conducting user research with people with disabilities activities is something we discussed a few years back in RDWG, as part of a wider effort to encourage researchers to integrate accessibility into their work.
>
> Before RDWG closed, we drafted an outline for a Tips for Accessibility-Aware Research resource:
> https://www.w3.org/WAI/RD/wiki/Tips_for_Accessibility-Aware_Research
>
> After the rechartering changes, the intention is that development of this work would continue under the proposed Research and Development Task Force (RDTF) of the APA Working Group:
> http://www.w3.org/WAI/RD/work-statement.html
>
> As of now, I’m not clear where things lie with the establishment of RDTF, and as yet I’ve not had an opportunity to become more involved. But it seems that the timing’s right to get moving with producing this guide for researchers, and the points raised in this discussion need to be incorporated.
>
> So I’d be keen to pick up work on developing this guide, and will investigate the best way of expediting progress.
>
> Dave
>
> PS I’d like to make clear that my previous comments on this topic weren’t intended to question the ethics of any work that’s already been done in this area. Gathering perspectives from a range of individuals is valuable in helping identify potential issues to investigate more formally. But clearly, if we’re to gather reliable data that can be used for significant decision-making, this requires larger, more controlled studies, and so the ethics considerations change.
>
>
>> On 11 Jul 2016, at 14:15, lisa.seeman <lisa.seeman@zoho.com> wrote:
>>
>> Hi Josh
>>
>> This is a great thread. I think an issue paper on user testing for COGA use groups would be a good idea. Not having any user testing with people with cognitive disabilities might have the result that they will never have fully useable services.
>>
>> I have not done a lot of testing, but a technique I have used is to avoid any language that suggests the user fails. For example, "can you see how to buy shoes" becomes "did the designer make it clear how to buy shoes". In other words, the implication is the design is at fault not the user. Limiting the amount of fail question to two, however they have been rephrased, is probably a good idea as well. I would always end with asking them to do tasks they can manage.
>>
>> Of course, informed consent is always a requirement.
>>
>> All the best
>> Lisa
>>
>>
>>
>>
>>
>>
>>
>>
>> ---- On Mon, 11 Jul 2016 13:14:52 +0300 <josh@interaccess.ie> wrote ----
>> [Chair hat off]
>>   
>> In a previous thread the issue of user testing with people with cognitive impairments has been brought up. I thought this could be a good time to
>> share some of my own thoughts in this area. As some of you may know, I ran a user testing lab in the National Council for the Blind of Ireland for around 10 years. During that time I ran user tests with a wide range of users, not just those who are blind/VIP. In that time I did a small amount of testing with people with cognitive impairments, mental health issues etc. I found this a difficult group to test with due to my own concern about the ethics of doing so properly.
>>   
>> One very strong reservation I have about this whole area is simple. Does the user have the ability the objectively separate the tasks they are  asked to perform in a test (and the natural success/failure when trying to completing these tasks) from their overall 'sense of self'? What I mean is that will the user be able to realise that their actions are being objectively observed without any 'judgement' on their performance?
>>   
>>   I would hate to think that a user would come away from a user test, where many tasks were failed (which is great usability information) but feeling worse about themselves, or as if _they_ were some kind of failure. In short, I think user testing is a bit of a performance, within an utterly contrived environment. Some people take to this well, others don't.
>>   
>> This is a very thorny issue but one I want to flag. I don't think testing should take place at all without a strong framework about how to deal with these sensitive situations. Sometimes you may have to make a call not to test, if it isn't in the best interest of the user test participant, even if they may be a perfect candidate for 'rich data'. I've made this call not to test in the past, and my overall take away was that I am largely uncomfortable doing this kind of testing, unless I'm sure it is in the participants best interest, separately from whatever the secondary need of a project/client is.
>>   
>> Thoughts?
>>   
>> Josh
>>
>>
> David Sloan
>
> UX Research Lead
> The Paciello Group
> dsloan@paciellogroup.com
>
> CONFIDENTIALITY NOTICE: The information contained in this message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any use, disclosure, dissemination, distribution or copying of any portion of this message or any attachment is strictly prohibited. If you think you have received this message in error, please notify the sender at the above e-mail address, and delete this e-mail along with any attachments. Thank you.
>
>