Re: This is significantly different from what was agreed. - was Re: CFC: Manual testing processes

TL;DR:

I will strongly oppose any Success Criteria that insists on user/usability
testing (even for exemptions), as the very nature of that testing is
subjective, and thus non-repeatable at scale.

*****

> For example use active voicing unless user testing with five people with
cognitive disabilities has shown passive voicing to be clearer.

Hi Lisa,

I now note that you have attached a quantity - *5* People with Cognitive
Disabilities - to this exception that is not in the Draft SC. Is that a
proposed edit to the emergent Success Criteria?

While I can understand this desire, the problem here is that it is next to
impossible to turn this into a measurable metric. Two teams of 5 testers
could emerge with 2 different 'responses' from user-testing (one thumbs-up,
the other thumbs-down) and it would be impossible to verify who would be
correct.

I also remain concerned over the definition of "clearer", as again this is
not a measurable metric.

Additionally, for compliance and monitoring tracking, how could this be
recorded? Will corporate entities be required to maintain those
user-testing logs in case they are challenged in court? That might sound
extreme, it is however also a very legitimate scenario (and question), and
proceeding with this clause in the SC will, I suspect, be the Achilles heal
and will ultimately result in this SC being rejected.

Requirements need to be repeatable, testable, and achievable on both small
boutique sites, as well as the largest multi-national, multi-language
sites. Demanding of large corporate clients that *all* corporate editorial
content must be reviewed by 5 disabled users before claiming conformance
(or an exception/exemption) will meet significant resistance, up-to and
including non-conformance (see my earlier note about 'Undue Burden').

JF




On Wed, Feb 15, 2017 at 8:07 AM, lisa.seeman <lisa.seeman@zoho.com> wrote:

> This is hard to explain by email.,
>
> We agreed that we were not going to make disability testing a blanket
> requirement for conformance.
> However in some cases that is not the same as what Andrew is saying -  "that
> if the only way to test a success criteria is to conduct user testing, then
> that is not “testable” with regard to WCAGT 2.1. "
>
>
> For example, we have a SC that requires you to use active voicing on
> critical information - but we have an exception:
>
> *Exceptions:*
>
>    - When a passive voice or a tense (other than present tense) is
>    clearer. Other voices or tenses may be used when it has been shown, via
>    user testing, to be easier to understand, friendlier, or appropriate.
>
> (In fact we have many exceptions but this is the one that is relevant here)
>
> Here we allow user testing to activate an exception.
>
> Is that still OK with the implication that Andrew has mentioned for this
> resolution? This is unclear to me. If it is it increase the scope of what
> was agreed.
>
>
> All the best
>
> Lisa Seeman
>
> LinkedIn <http://il.linkedin.com/in/lisaseeman/>, Twitter
> <https://twitter.com/SeemanLisa>
>
>
>
>
> ---- On Wed, 15 Feb 2017 15:22:51 +0200 *Joshue O
> Connor<josh@interaccess.ie <josh@interaccess.ie>>* wrote ----
>
> >The resolution implication is different to what was discussed. We CAN NOT
> pass the >resolution if this implication does not allow for exceptions via
> user testing at least >without a real discussion so we all understand what
> is at stake
>
> Sorry Lisa. I don't understand your point here. If anyone does user
> testing and finds issues for any given target group then they are totally
> free to flag those issues and make recommendations on the outputs or
> results of a test.
>
> There is no explicit restriction here that I can see, can you clarify what
> you mean?
>
> Thanks
>
> Josh
>
> lisa.seeman <lisa.seeman@zoho.com>
> 15 February 2017 at 13:15
> The resolution implication is different to what was discussed. We CAN NOT
> pass the resolution if this implication does not allow for exceptions via
> user testing at least without a real discussion so we all understand what
> is at stake
>
> We agreed we were not making user testing a requirement for conformance.
>
> This implication is significantly different and changes things.
>
> User testing was ok to enable an exception. In other words it is not
> required, but you can claim an exception via use testing.
> For example use active voicing unless user testing with five people with
> cognitive disabilities has shown passive voicing to be clearer.
>
> This implication has not been discussed . The vote is meaningless if this
> "implication" has nt been fully understood by everyone voting
>
> This add be shown to be
>
> All the best
>
> Lisa Seeman
>
> LinkedIn <http://il.linkedin.com/in/lisaseeman/>, Twitter
> <https://twitter.com/SeemanLisa>
>
>
>
>
> ---- On Wed, 15 Feb 2017 08:59:19 +0200 *
> Chakravarthula<srchakravarthula@informatica.com>
> <srchakravarthula@informatica.com>* wrote ----
>
>
> Chakravarthula, Srinivasu <srchakravarthula@informatica.com>
> 15 February 2017 at 06:59
> +1
>
> Regards,
> Srinivasu Chakravarthula | Informatica | @CSrinivasu
> Sent from my iPhone
>
> Andrew Kirkpatrick <akirkpat@adobe.com>
> 14 February 2017 at 03:19
> Call For Consensus — ends Wednesday February 15th at 10:30pm Boston time.
>
> The requirements for WCAG 2.1 SC's (https://www.w3.org/WAI/GL/
> wiki/WCAG_2.1_Success_Criteria have been discussed at length. Included in
> the requirements is #2, which states "Be testable through automated or
> manual processes”, which indicates that in order for a success criteria to
> be regarded as “testable” it must be possible to determine whether a page
> passes that criteria using automated or manual testing processes.
>
> On last Tuesday’s call the WG came to a resolution regarding this item,
> specifically related to user testing. The group also surveyed this
> question, and arrived at a unanimous agreement:
>
> "User testing is not a required part of a manual testing process for WCAG
> test criteria.”
>
> This resolution indicates that if the only way to test a success criteria
> is to conduct user testing, then that is not “testable” with regard to
> WCAGT 2.1.
>
> The Working Group will recommend strongly in WCAG 2.1 (as it did in WCAG
> 2.0) that user testing be conducted.
>
> For background:
> Call minutes: http://www.w3.org/2017/02/07-ag-minutes.html
> Survey: https://www.w3.org/2002/09/wbs/35422/testing20170207/results#xq1
>
> If you have concerns about this proposed consensus position that have not
> been discussed already and feel that those concerns result in you “not
> being able to live with” this decision, please let the group know before
> the CfC deadline.
>
> Thanks,
> AWK
>
> Andrew Kirkpatrick
> Group Product Manager, Standards and Accessibility
> Adobe
>
> akirkpat@adobe.com
> http://twitter.com/awkawk
>
>
> --
> Joshue O Connor
> Director | InterAccess.ie
>
>
>
>


-- 
John Foliot
Principal Accessibility Strategist
Deque Systems Inc.
john.foliot@deque.com

Advancing the mission of digital accessibility and inclusion