Re: Automated and manual testing process

>Authors and reviewers need more concrete and specific criteria than
whether they think people with a broad range of learning/cognitive
abilities would understand it  not a question that one can reliably answer
unless one is a specialist in cognition, I suspect

I would add that any metrics we come up with should have high "inter
relater reliability" between specialists in cognition, which may not be
easy because of the range of symptoms.
Established best practices and unity among experts is key.  Its easier for
us to formulate testable statements under those conditions.


Cheers,
David MacDonald



*Can**Adapt* *Solutions Inc.*

Tel:  613.235.4902

LinkedIn
<http://www.linkedin.com/in/davidmacdonald100>

twitter.com/davidmacd

GitHub <https://github.com/DavidMacDonald>

www.Can-Adapt.com <http://www.can-adapt.com/>



*  Adapting the web to all users*
*            Including those with disabilities*

If you are not the intended recipient, please review our privacy policy
<http://www.davidmacd.com/disclaimer.html>

On Mon, Jan 30, 2017 at 3:57 PM, White, Jason J <jjwhite@ets.org> wrote:

>
>
>
>
>
>
> *From:* Michael Pluke [mailto:Mike.Pluke@castle-consult.com
> <Mike.Pluke@castle-consult.com>]
> *Sent:* Monday, January 30, 2017 1:21 PM
>
> Wherever possible the COGA Task Force has tried to propose SCs that do not
> rely on subjective testing, but automatically assessing whether, for
> example, a label accurately and clearly describes the thing that it labels
> in a way that users with learning disabilities might be able to understand
> is currently not something that is easy to automate. For such cases,
> subjective testing will be the only practical way to assess whether a
> significant accessibility barrier exists.
>
> *[Jason] Can you offer criteria to be used in making the judgment that
> would help to achieve reliable results across evaluators?*
>
>
>
> *I think there are two distinct issues here. The first concerns
> automation, which I agree is largely irrelevant in these cases except to
> the extent to which measures of linguistic complexity serve as useful
> guides (e.g., in characterizing the lower secondary education level as in
> 3.1.5). The second issue concerns reliability of informed human
> evaluations, and whether those evaluations distinguish adequately between
> content that is more and content that is less accessible to people with
> learning and cognitive disabilities.*
>
>
>
> To evaluate the adequacy of a label, as in your example, I can check
> whether I think it unambiguously identifies the thing labelled, whether the
> vocabulary used appears in lists of commonly understood words in the
> language (substituting an alternative that uses such words, if this remains
> unambiguous regarding what the label’s purpose is), whether it uses
> vocabulary associated with the relevant discipline/subject-matter, if
> applicable, and various other criteria that might be appropriate. The
> problem is to provide the right guidance to evaluators and authors as to
> what they should be checking for. Authors and reviewers need more concrete
> and specific criteria than whether they think people with a broad range of
> learning/cognitive abilities would understand it (not a question that one
> can reliably answer unless one is a specialist in cognition, I suspect).
> There is also the role to be played by the author’s intended/assumed
> audience, which shouldn’t be so defined as to exclude people with
> disabilities who have the right skills, background and education, for
> example, to participate in the activity or read the material.
>
>
>
> ------------------------------
>
> This e-mail and any files transmitted with it may contain privileged or
> confidential information. It is solely for use by the individual for whom
> it is intended, even if addressed incorrectly. If you received this e-mail
> in error, please notify the sender; do not disclose, copy, distribute, or
> take any action in reliance on the contents of this information; and delete
> it from your system. Any other use of this e-mail is prohibited.
>
> Thank you for your compliance.
> ------------------------------
>