Re: Automated and manual testing process

Hi Andrew,

I have found myself splitting accessibility testing into two areas when talking about it, in essence:


1)       Technical Accessibility Testing = Continuous code / DOM level – Automatic, Guided Automatic, Manual testing to determine quantitative things. For example, does an img element in the current DOM-state have a mechanism that enables an accessible name to be calculated.  With:

a.       The creation of immensely atomic tests in order to reduce the need for expert human judgement, whenever possible; and

b.       Now, Machine Learning-based tests deployed through modern software testing frameworks to enable automatic testing in areas that have to-date been perceived as only being assessable by humans (e.g. determining if an alternative text value is relevant for an image, in the image’s determined context).


2)       Usability testing by people with disabilities = Scheduled testing conducted by real users (with disabilities; using AT), to provide feedback on qualitative aspects of software (in-build, then live) from their accessibility perspectives.  For example, is the calculated accessible name meaningful / good; is a page structured in a useful manner; can the carousel widget actually be used by a keyboard, etc…

Thought I’d just throw it into the mix.  Hope this helps.

Very best regards

Alistair

---

Alistair Garrison
Director of Accessibility Research
SSB Bart Group
Visit us online: Website<http://www.ssbbartgroup.com/> | Twitter<https://twitter.com/SSBBARTGroup> | Facebook<https://www.facebook.com/ssbbartgroup> | LinkedIn<https://www.linkedin.com/company/355266?trk=tyah> | Blog<http://www.ssbbartgroup.com/blog/>

From: Andrew Kirkpatrick <akirkpat@adobe.com>
Date: Saturday, 28 January 2017 at 22:36
To: WCAG <w3c-wai-gl@w3.org>
Subject: Automated and manual testing process
Resent-From: <w3c-wai-gl@w3.org>
Resent-Date: Saturday, 28 January 2017 at 22:37

AGWGer’s,
I’d like to get the thoughts from the group on what constitutes “manual testing” (I’m more comfortable with what counts as automated testing).

Testing the presence of alternative text on an image in HTML or other formats can be done with automated testing, but testing for the presence of good alternative text requires (at least for now) human involvement in the test process (manual testing).

What if testing cannot be done by a single person and requires user testing – does that count as manual testing, or is that something different?

Thanks,
AWK

Andrew Kirkpatrick
Group Product Manager, Standards and Accessibility
Adobe

akirkpat@adobe.com
http://twitter.com/awkawk