Re: Remote usability testing with disabled people

what an excellent set of questions.

One place I would start is  “TeachAccess.org <http://teachaccess.org/>”.  This is a new effort that is just started to build — but they are trying to pull things from diverse places.   
We are also creating a repository for this type of information in the RESOURCE section of our new DeveloperSpace.   It will be launched this summer

   
Teaching people to be great testers though is more an apprentice thing than an online course kind of thing.     So I’m really interested (and so will Teach Access) in what you think about all this and how we can create better training for accessibility evaluators — including mentorship/apprenticeship opportunities. 


Gregg C Vanderheiden
greggvan@umd.edu



> On Feb 27, 2017, at 8:43 AM, Juliette <piazza.juliette@gmail.com> wrote:
> 
> Hello
> Thank a lot Bob, Gregg, Wayne, Katie, Tom, Lars, Andrew, Beranek, Sean and Morten for your input. Below, I am summarizing what you said and I am adding some questions. Feel free to correct me if I am wrong.
> 
> We are talking about two kinds of testings with two types of profiles.
> 
> 1: Accessibility compliance testing
> Goal: Determine whether a solution can be used by as many people as possible, including people with a variety of disabilities.Example: can a solution be used without a mouse, with magnification or a screen reader? 
> Tester profile and considerations:
> Expert. That means they have proficient IT and AT skills. Otherwise, it won’t be reliable, it will be expensive and time-consuming.
> The exact technologies they are using need to be identified.
> Challenges: How do you train such testers? Are there any free resources available? Are there any free courses?
> 
>  2:(Extended) Usability testing
> Goal:  The tester should be describing why they are having issues in generic terms and the test coordinator should be probing to ensure they understand the barriers. 
> Tester profile and considerations:
> IT skills from poor skills to advance. Users who have the most trouble with technology and people with disabilities need to be included.
> Platforms and AT they are using
> Testers need to have a good understanding of the related product and context
> Challenges: This is more related to remote testing challenges. How do you validate the user is performing the set of tests you want and they understand the related product? Such as:
> How do you monitor their methodology of using the page or sub-section of the page/UI?
> How do you track their method of using the page with their related technology?
> How do you verify the issue is related to the page, their usage of the assistive technology or their technical skill set?
> If a user finds an issue and they cannot clearly explain the problem what method is used to confirm if the issue is valid
> Additional questions:
> 
> How do you evaluate the IT skill level of a tester? Is there any ‘standards’ defining the IT skill level of a person? 
> Sean you said: “ I can see companies using the service to ask for a specific style of tester.” What are these companies expecting from such specific testers? Would they need to be expert or novice testers?
> Morten, my understanding is that you recruit your testers, you train them and then they do the testing and the report. Did you experience, recruiting testers with low digital skills? 
> For those who are attending the CSUN, have a good event!
> Kind Regards,
> 
> Juliette
> 
> 
> 
> On 27 February 2017 at 05:06, Andrew Arch <andrew.arch@digital.gov.au <mailto:andrew.arch@digital.gov.au>> wrote:
> Good discussion here - the issue of recruiting people with disability for user research interviews and for usability and accessibility testing is a common one.
> 
> Just to add to Sean's point about would they "understand the related product" - the banking systems, government systems, etc are very different round the world and I think anyone doing remote testing of many sites and on-line services would also want to be sure the users came to the testing from the right understanding and context.
> 
> Andrew
> 
> -------------------
> Andrew Arch
> Accessibility & Inclusivity Lead 
>  Digital Transformation Agency (DTA)
> Australian Government
> www.dta.gov.au <http://www.dta.gov.au/> 
> p. +61 (0)428 134 529 t. @DTA <https://twitter.com/DTA>  |  @amja
>  <https://twitter.com/amja>
> 
> On 27 February 2017 at 14:38, <accessys@smart.net <mailto:accessys@smart.net>> wrote:
> 
> very good points
> 
> also we shuld also remember to be sure to include multiple platforms from the latest windows to Uinx mainframes (who would probably be skilled AT/IT people) and everything used around the world.in <http://world.in/> betweeen
> 
> Bob
> 
> 
> 
> 
> On Mon, 27 Feb 2017, Sean Murphy (seanmmur) wrote:
> 
> Date: Mon, 27 Feb 2017 01:09:07 +0000
> From: "Sean Murphy (seanmmur)" <seanmmur@cisco.com <mailto:seanmmur@cisco.com>>
> To: Juliette <piazza.juliette@gmail.com <mailto:piazza.juliette@gmail.com>>,
>     "w3c-wai-ig@w3.org <mailto:w3c-wai-ig@w3.org>" <w3c-wai-ig@w3.org <mailto:w3c-wai-ig@w3.org>>
> Subject: RE: Remote usability testing with disabled people
> Resent-Date: Mon, 27 Feb 2017 01:09:49 +0000
> Resent-From: w3c-wai-ig@w3.org <mailto:w3c-wai-ig@w3.org>
> 
> 
> Julie and all,
> 
> Excellent discussion and very valid points. I agree with the statement Greg outlined. You need a combination of both. A end-user tester should be describing why they are having issues in generic terms and the test coordinator should be probing to ensure they understand the barriers. The user pool should be ranging from people with poor skills to advance. Stating the obvious, everyone has different technical skills with computers. The other pool is of those who have accessibility skills.  Again, there will be a range of expertise here as well. Identifying the different disabilities is a sub-set of the relevant pools.
> 
> Julie’s and Greg’s approach of having voluntary people in a database that can do face-to-face or remote testing internationally is a good idea. I can see companies using the service to ask for a specific style of tester. The profiles of these users would have to be built to make this work and this is the true challenge. How do you determine a person’s skill set?
> 
> 
> In relation to remote testing for a generic user, how do you validate the user is performing the set of tests you want and they understand the related product? Such as:
> 
> ·        How do you monitor their methodology of using the page or sub-section of the page/UI?
> 
> ·        How do you track their method of using the page with their related technology?
> 
> ·        How do you verify the issue is related to the page, their usage of the assistive technology  or their technical skill set?
> 
> ·        If a user finds an issue and they cannot clearly explain the problem what method is used to confirm if the issue is valid? This from my personal experience is the most challenging. As the average user doesn’t understand the terminology or care. So if they come across a problem, they might not explain it in an effective means with the risk of the issue being dismissed.
> 
> Still this is a good concept and I am very interested in assisting or finding out more.
> 
> 
> 
> Sean Murphy
> Accessibility Software engineer
> seanmmur@cisco.com <mailto:seanmmur@cisco.com>
> Tel: +61 2 8446 7751 <tel:%2B61%202%208446%207751>       Cisco Systems, Inc.
> The Forum 201 Pacific Highway
> ST LEONARDS
> 2065
> Australia
> cisco.com <http://cisco.com/>
> Think before you print.
> This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
> 
> From: Juliette [mailto:piazza.juliette@gmail.com <mailto:piazza.juliette@gmail.com>]
> Sent: Sunday, 26 February 2017 12:19 AM
> To: w3c-wai-ig@w3.org <mailto:w3c-wai-ig@w3.org>
> Subject: Remote usability testing with disabled people
> 
> Hello,
> 
> I launched,  Inclusight<http://www.inclu-sight/ <http://www.inclu-sight/>>, a startup that provides disabled participants for user testings. After providing for a while, disabled participants for face-to-face user testing, I figured out this was not the best solution. It's not convenient at all for disabled people as they need to travel and to plan the session a long time in advance. And when they start the testing, they figure out they cannot use their own familiar configurations. It's also a pain for user researchers who, on top of that, are not always aware of how is it to work with disabled people.
> That's how I came up with the ambition of offering remote usability testings for disabled people. At this stage, I am looking for professionals willing to share with me their experience in doing remote user testing with vulnerable or disabled people. I want to understand how you could make the most benefit from Inclusight.
> I am looking forward to hearing from user researchers, web accessibility experts or any other professionals.
> Kind Regards,
> --
> Juliette
> 
> 
> 
> 
> 
> 
> 
> -- 
> Juliette