RE: Automated and manual testing process

From: Michael Pluke [mailto:Mike.Pluke@castle-consult.com]
Sent: Monday, January 30, 2017 1:21 PM

Wherever possible the COGA Task Force has tried to propose SCs that do not rely on subjective testing, but automatically assessing whether, for example, a label accurately and clearly describes the thing that it labels in a way that users with learning disabilities might be able to understand is currently not something that is easy to automate. For such cases, subjective testing will be the only practical way to assess whether a significant accessibility barrier exists.
[Jason] Can you offer criteria to be used in making the judgment that would help to achieve reliable results across evaluators?

I think there are two distinct issues here. The first concerns automation, which I agree is largely irrelevant in these cases except to the extent to which measures of linguistic complexity serve as useful guides (e.g., in characterizing the lower secondary education level as in 3.1.5). The second issue concerns reliability of informed human evaluations, and whether those evaluations distinguish adequately between content that is more and content that is less accessible to people with learning and cognitive disabilities.

To evaluate the adequacy of a label, as in your example, I can check whether I think it unambiguously identifies the thing labelled, whether the vocabulary used appears in lists of commonly understood words in the language (substituting an alternative that uses such words, if this remains unambiguous regarding what the label’s purpose is), whether it uses vocabulary associated with the relevant discipline/subject-matter, if applicable, and various other criteria that might be appropriate. The problem is to provide the right guidance to evaluators and authors as to what they should be checking for. Authors and reviewers need more concrete and specific criteria than whether they think people with a broad range of learning/cognitive abilities would understand it (not a question that one can reliably answer unless one is a specialist in cognition, I suspect). There is also the role to be played by the author’s intended/assumed audience, which shouldn’t be so defined as to exclude people with disabilities who have the right skills, background and education, for example, to participate in the activity or read the material.


________________________________

This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.


Thank you for your compliance.

________________________________