Re: Remote usability testing with disabled people

Hello

Thank a lot Bob, Gregg, Wayne, Katie, Tom, Lars, Andrew, Beranek, Sean and
Morten for your input. Below, I am summarizing what you said and I am
adding some questions. Feel free to correct me if I am wrong.

We are talking about two kinds of testings with two types of profiles.


1: Accessibility compliance testing

Goal: Determine whether a solution can be used by as many people as
possible, including people with a variety of disabilities.Example: can a
solution be used without a mouse, with magnification or a screen reader?

Tester profile and considerations:

   - Expert. That means they have proficient IT and AT skills. Otherwise,
   it won’t be reliable, it will be expensive and time-consuming.
   - The exact technologies they are using need to be identified.

*Challenges:* How do you train such testers? Are there any free resources
available? Are there any free courses?


2:(Extended) Usability testing

*Goal:*  The tester should be describing why they are having issues in
generic terms and the test coordinator should be probing to ensure they
understand the barriers.

*Tester profile and considerations:*

   - IT skills from poor skills to advance. Users who have the most trouble
   with technology and people with disabilities need to be included.
   - Platforms and AT they are using
   - Testers need to have a good understanding of the related product and
   context

*Challenges:* This is more related to remote testing challenges. How do you
validate the user is performing the set of tests you want and they
understand the related product? Such as:

   - How do you monitor their methodology of using the page or sub-section
   of the page/UI?
   - How do you track their method of using the page with their related
   technology?
   - How do you verify the issue is related to the page, their usage of the
   assistive technology or their technical skill set?
   - If a user finds an issue and they cannot clearly explain the problem
   what method is used to confirm if the issue is valid

*Additional questions*:


   - How do you evaluate the IT skill level of a tester? Is there any
   ‘standards’ defining the IT skill level of a person?
   - Sean you said: “ I can see companies using the service to ask for a
   specific style of tester.” What are these companies expecting from such
   specific testers? Would they need to be expert or novice testers?
   - Morten, my understanding is that you recruit your testers, you train
   them and then they do the testing and the report. Did you experience,
   recruiting testers with low digital skills?

For those who are attending the CSUN, have a good event!
Kind Regards,

Juliette


On 27 February 2017 at 05:06, Andrew Arch <andrew.arch@digital.gov.au>
wrote:

> Good discussion here - the issue of recruiting people with disability for
> user research interviews and for usability and accessibility testing is a
> common one.
>
> Just to add to Sean's point about would they "understand the related
> product" - the banking systems, government systems, etc are very
> different round the world and I think anyone doing remote testing of many
> sites and on-line services would also want to be sure the users came to the
> testing from the right understanding and context.
>
> Andrew
>
> -------------------
>
> Andrew Arch
>
> Accessibility & Inclusivity Lead
>
> Digital Transformation Agency (DTA)
>
> Australian Government
>
> www.dta.gov.au
> p. +61 (0)428 134 529 t. @DTA <https://twitter.com/DTA>  |  @amja
> <https://twitter.com/amja>
>
> On 27 February 2017 at 14:38, <accessys@smart.net> wrote:
>
>>
>> very good points
>>
>> also we shuld also remember to be sure to include multiple platforms from
>> the latest windows to Uinx mainframes (who would probably be skilled AT/IT
>> people) and everything used around the world.in betweeen
>>
>> Bob
>>
>>
>>
>>
>> On Mon, 27 Feb 2017, Sean Murphy (seanmmur) wrote:
>>
>> Date: Mon, 27 Feb 2017 01:09:07 +0000
>>> From: "Sean Murphy (seanmmur)" <seanmmur@cisco.com>
>>> To: Juliette <piazza.juliette@gmail.com>,
>>>     "w3c-wai-ig@w3.org" <w3c-wai-ig@w3.org>
>>> Subject: RE: Remote usability testing with disabled people
>>> Resent-Date: Mon, 27 Feb 2017 01:09:49 +0000
>>> Resent-From: w3c-wai-ig@w3.org
>>>
>>>
>>> Julie and all,
>>>
>>> Excellent discussion and very valid points. I agree with the statement
>>> Greg outlined. You need a combination of both. A end-user tester should be
>>> describing why they are having issues in generic terms and the test
>>> coordinator should be probing to ensure they understand the barriers. The
>>> user pool should be ranging from people with poor skills to advance.
>>> Stating the obvious, everyone has different technical skills with
>>> computers. The other pool is of those who have accessibility skills.
>>> Again, there will be a range of expertise here as well. Identifying the
>>> different disabilities is a sub-set of the relevant pools.
>>>
>>> Julie’s and Greg’s approach of having voluntary people in a database
>>> that can do face-to-face or remote testing internationally is a good idea.
>>> I can see companies using the service to ask for a specific style of
>>> tester. The profiles of these users would have to be built to make this
>>> work and this is the true challenge. How do you determine a person’s skill
>>> set?
>>>
>>>
>>> In relation to remote testing for a generic user, how do you validate
>>> the user is performing the set of tests you want and they understand the
>>> related product? Such as:
>>>
>>> ·        How do you monitor their methodology of using the page or
>>> sub-section of the page/UI?
>>>
>>> ·        How do you track their method of using the page with their
>>> related technology?
>>>
>>> ·        How do you verify the issue is related to the page, their usage
>>> of the assistive technology  or their technical skill set?
>>>
>>> ·        If a user finds an issue and they cannot clearly explain the
>>> problem what method is used to confirm if the issue is valid? This from my
>>> personal experience is the most challenging. As the average user doesn’t
>>> understand the terminology or care. So if they come across a problem, they
>>> might not explain it in an effective means with the risk of the issue being
>>> dismissed.
>>>
>>> Still this is a good concept and I am very interested in assisting or
>>> finding out more.
>>>
>>>
>>>
>>> Sean Murphy
>>> Accessibility Software engineer
>>> seanmmur@cisco.com
>>> Tel: +61 2 8446 7751       Cisco Systems, Inc.
>>> The Forum 201 Pacific Highway
>>> ST LEONARDS
>>> 2065
>>> Australia
>>> cisco.com
>>> Think before you print.
>>> This email may contain confidential and privileged material for the sole
>>> use of the intended recipient. Any review, use, distribution or disclosure
>>> by others is strictly prohibited. If you are not the intended recipient (or
>>> authorized to receive for the recipient), please contact the sender by
>>> reply email and delete all copies of this message.
>>>
>>> From: Juliette [mailto:piazza.juliette@gmail.com]
>>> Sent: Sunday, 26 February 2017 12:19 AM
>>> To: w3c-wai-ig@w3.org
>>> Subject: Remote usability testing with disabled people
>>>
>>> Hello,
>>>
>>> I launched,  Inclusight<http://www.inclu-sight/>, a startup that
>>> provides disabled participants for user testings. After providing for a
>>> while, disabled participants for face-to-face user testing, I figured out
>>> this was not the best solution. It's not convenient at all for disabled
>>> people as they need to travel and to plan the session a long time in
>>> advance. And when they start the testing, they figure out they cannot use
>>> their own familiar configurations. It's also a pain for user researchers
>>> who, on top of that, are not always aware of how is it to work with
>>> disabled people.
>>> That's how I came up with the ambition of offering remote usability
>>> testings for disabled people. At this stage, I am looking for professionals
>>> willing to share with me their experience in doing remote user testing with
>>> vulnerable or disabled people. I want to understand how you could make the
>>> most benefit from Inclusight.
>>> I am looking forward to hearing from user researchers, web accessibility
>>> experts or any other professionals.
>>> Kind Regards,
>>> --
>>> Juliette
>>>
>>>
>>>
>>
>


-- 
Juliette