Re: Automated and manual testing process

WCAG actually doesn't use the word "Manual testing" The phrase in the
introduction is "...  to be testable with a combination of automated
testing and human evaluation."

Here is the paragraph

============
All WCAG 2.0 Success Criteria are written as testable criteria for
objectively determining if content satisfies them. Testing the Success
Criteria would involve a combination of automated testing and human
evaluation. The content should be tested by those who understand how people
with different types of disabilities use the Web.

Testing and testable in the context refer to functional testing, that is
verifying that the content functions as expected, or in this case, that it
satisfies the Success Criteria. Although content may satisfy all Success
Criteria, the content may not always be usable by people with a wide
variety of disabilities. Therefore, usability testing is recommended, in
addition to the required functional testing. Usability testing aims to
determine how well people can use the content for its intended purpose. It
is recommended that users with disabilities be included in test groups when
performing usability testing.

https://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html
=============


Mike says:
>> for example, a label accurately and clearly describes the thing that it
labels in a way that users with learning disabilities might be able to
understand is currently not something that is easy to automate. For such
cases, subjective testing will be the only practical way to assess whether
a significant accessibility barrier exists.

If we can establish a metric that is reliable and that we can establish a
high "inter rater reliability" among human testers who "derstand how people
with different types of disabilities use the Web" then it is testable. In
that case it's not "subjective" in the eyes of WCAG, and can be formulated
as a testable statement.

Cheers,
David MacDonald



*Can**Adapt* *Solutions Inc.*

Tel:  613.235.4902

LinkedIn
<http://www.linkedin.com/in/davidmacdonald100>

twitter.com/davidmacd

GitHub <https://github.com/DavidMacDonald>

www.Can-Adapt.com <http://www.can-adapt.com/>



*  Adapting the web to all users*
*            Including those with disabilities*

If you are not the intended recipient, please review our privacy policy
<http://www.davidmacd.com/disclaimer.html>

On Mon, Jan 30, 2017 at 9:49 AM, Bailey, Bruce <Bailey@access-board.gov>
wrote:

> Andrew asked:
> > What if testing cannot be done by a single person and requires user
> testing - does that count as manual testing, or is that something different?
>
> I agree with folks pointing out that "manual testing" is not a great
> term.  Alastair clarified what the term means for him, and I think it is
> more than good enough:
>
> >> Manual review/audit, where an expert goes through a sample of pages
> using the guidelines. This can assess 'appropriateness' of things like alt
> text, headings,  markup and interactions (e.g. scripted events).
>
> That *can* be a single person, but I do not trust for myself.  IMHO the
> fastest and best way to test is with an expert using screen reading
> software, while a sighted expert looks over their shoulder to ensure that
> the UI is not failing in a way that causes something important to be
> missed.  So I think that hits what AWK "cannot be done by a single person
> and requires user testing".
>
> But all the WCAG 2.0 failures can be tracked back to source code, and we
> absolutely want to keep that level of objectivity.  So what I characterized
> as the fastest is best way to test is not the only way to test.
>
> Alastair continues:
> >> If something 'requires' multiple testers then we need to (try to) write
> the guideline or guidance better.
>
> To which I can only say:
> +1
>
>