Re: CFC: Manual testing processes

For reference


In WCAG 2.0 WG     
We always encouraged developers to have users test their site — a wide variety of users.  It is really a great way to find problems in a site that miss inspection.  Especially egregious (show stopper) ones. 
But user testing was never required or used as a method for an author to know they had met an SC - for all of the reasons cited below and then some.  





Gregg C Vanderheiden
greggvan@umd.edu



> On Feb 16, 2017, at 10:32 AM, Katie Haritos-Shea GMAIL <ryladog@gmail.com> wrote:
> 
> All, <>
>  
> It is only “beyond the scope” because of the restrictive timeline that we allowed to be imposed upon us. 
>  
> For myself, I would have liked to see us address the issues of Cognitive and Low Vision disabilities to the full breadth of the need of these communities. Great work is being done, but this does make me sad and frustrated. I am not 100% sure that user testing would be the *only* way to test some of the identified issues, but the fact is, we will never know, because we let arbitrary timelines control our output and content. 
>  
> For now, we do the best we can, and should apologize to the communities we will have left underserved when we release……
>  
> Meanwhile, I as an AC Rep, will do what I can to inform the membership about the importance differences between Accessibility specs and background browser protocol, language and API specs – and how the thresholds on timelines need to reflect those difference – to produce the most useful and effective solutions for people with disabilities trying to use the web.
>  
> If I could change this path we are on, I would direct that the Mobile SC get released ASAP (on their own, un-numbered) in the various Review Draft forms, and when ready, be Identified officially as a Pre-cursor to WCAG 2.1 for Mobile Content, so people could start using them. In the interim we could then provide the Cognitive and Low Vision communities with the due diligence that is needed, and in the end release a 2.1 that addresses all needs (Mobile, Low Vision, Cognitive, ePublications).
>  
> ​​​​​* katie *
>  
> Katie Haritos-Shea 
> Principal ICT Accessibility Architect (WCAG/Section 508/ADA/AODA)
>  
> Cell: 703-371-5545 | ryladog@gmail.com <mailto:ryladog@gmail.com> | Oakton, VA | LinkedIn Profile <http://www.linkedin.com/in/katieharitosshea/> | Office: 703-371-5545 | @ryladog <https://twitter.com/Ryladog>
> 
> NOTE: The content of this email should be construed to always be an expression of my own personal independent opinion, unless I identify that I am speaking on behalf of Knowbility, as their AC Rep at the W3C - and - that my personal email never expresses the opinion of my employer, Deque Systems.
>  
> From: Joshue O Connor [mailto:josh@interaccess.ie] 
> Sent: Thursday, February 16, 2017 4:58 AM
> To: Wayne Dick <wayneedick@gmail.com>
> Cc: Andrew Kirkpatrick <akirkpat@adobe.com>; WCAG <w3c-wai-gl@w3.org>
> Subject: Re: CFC: Manual testing processes
>  
> I'm open to looking at including user testing and other inclusive design methodologies in AG 3.0 and do hear what you are saying Wayne but this is just beyond the scope of 2.1 IMO.
> 
> Josh
> 
> 
>> Wayne Dick <mailto:wayneedick@gmail.com>
>> 16 February 2017 at 05:00
>> -1
>> Our testing methodology has not proven to be as reliable as we think. I think there may be cases where user testing is necessary. 
>> We are not making success criteria to make life easy for developers or auditors. We are removing barriers. 
>> We have a year and a half before this is released. I think it is time for accessibility auditors to include user testing when no other reliable testing method exists to assess the presence of a barrier.
>> There are reliable small sample testing techniques. Accessibility auditors should learn them.
>> Wayne
>>  
>> 
>> Andrew Kirkpatrick <mailto:akirkpat@adobe.com>
>> 14 February 2017 at 03:19
>> Call For Consensus — ends Wednesday February 15th at 10:30pm Boston time.
>>  
>> The requirements for WCAG 2.1 SC's (https://www.w3.org/WAI/GL/wiki/WCAG_2.1_Success_Criteria <https://www.w3.org/WAI/GL/wiki/WCAG_2.1_Success_Criteria> have been discussed at length. Included in the requirements is #2, which states "Be testable through automated or manual processes”, which indicates that in order for a success criteria to be regarded as “testable” it must be possible to determine whether a page passes that criteria using automated or manual testing processes.
>>  
>> On last Tuesday’s call the WG came to a resolution regarding this item, specifically related to user testing. The group also surveyed this question, and arrived at a unanimous agreement:
>>  
>> "User testing is not a required part of a manual testing process for WCAG test criteria.”
>>  
>> This resolution indicates that if the only way to test a success criteria is to conduct user testing, then that is not “testable” with regard to WCAGT 2.1.
>>  
>> The Working Group will recommend strongly in WCAG 2.1 (as it did in WCAG 2.0) that user testing be conducted.
>>  
>> For background:
>> Call minutes: http://www.w3.org/2017/02/07-ag-minutes.html <http://www.w3.org/2017/02/07-ag-minutes.html>
>> Survey: https://www.w3.org/2002/09/wbs/35422/testing20170207/results#xq1 <https://www.w3.org/2002/09/wbs/35422/testing20170207/results#xq1>
>>  
>> If you have concerns about this proposed consensus position that have not been discussed already and feel that those concerns result in you “not being able to live with” this decision, please let the group know before the CfC deadline.
>>  
>> Thanks,
>> AWK
>>  
>> Andrew Kirkpatrick
>> Group Product Manager, Standards and Accessibility
>> Adobe 
>>  
>> akirkpat@adobe.com <mailto:akirkpat@adobe.com>
>> http://twitter.com/awkawk <http://twitter.com/awkawk> 
> -- 
> Joshue O Connor
> Director | InterAccess.ie