Re: Automated and manual testing process

Andrew,
Here is my take:

Automated testing involves  the use of a tool that  is able to review a complete page or a portion thereof and list specific failures against WCAG SCs.
Manual testing: involves  human testers identifying WCAG failures without relying on an automated tool.
Manual testing may involve testing with assistive technologies as well as use of other software tools that measure contrast or flickering rate or changing the browser or OS settings or doing code reviews. The human tester uses the results of such tests to determine specific instances of failures.
Human testers may rely on automated testing tools to identify certain failures efficiently and exhaustively based on ones experience of a tool. 
Automated tools may need to be tweaked by knowledgeable  human testers for specific (or types of) content to minimize reporting of false-positives.
I have always believed that it is a team process. It is difficult for one person to  identify all violations even when an automated tool is relied upon. In the field of usability too there is an article from 2000 (which I had referenced in my MS thesis)  that says that testing with five users uncovers most  issues.
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ 

Thanks,
Sailesh Panchang

--------------------------------------------
On Sat, 1/28/17, Andrew Kirkpatrick <akirkpat@adobe.com> wrote:

 Subject: Automated and manual testing process
 To: "WCAG" <w3c-wai-gl@w3.org>
 Date: Saturday, January 28, 2017, 5:36 PM
 
 
 
  
 
 
 AGWGer’s,
 I’d like to get the thoughts from the group on what
 constitutes “manual testing” (I’m more comfortable
 with what counts as automated testing).
 
 
 
 Testing the presence of alternative text on an image in
 HTML or other formats can be done with automated testing,
 but testing for the presence of
 good alternative text
 requires (at least for now) human involvement in the test
 process (manual testing).
 
 
 
 What if testing cannot be done by a single person and
 requires user testing – does that count as manual testing,
 or is that something different?
 
 
 
 
 
 Thanks,
 AWK
 
 
 
 Andrew Kirkpatrick
 Group Product Manager, Standards and
 Accessibility
 Adobe 
 
 
 
 akirkpat@adobe.com
 http://twitter.com/awkawk

Received on Saturday, 28 January 2017 23:57:55 UTC