Re: Measurable vs testable [was: Silver Requirements Issues]

Hi Jeanne,

I am the COGA Task Force manager for the Accessible Authentication SC. I am also the author of the Web Security and Privacy issue paper it is based on. See: https://rawgit.com/<https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>w3c<https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/<https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>coga<https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/master/issue-papers/privacy-security.html<https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>

Accessible Authentication was the only COGA SC considered at Level A for 2.1. That it did not make it was due in part to the testability issues you and Alastair are discussing. However, the push back involved several other objections.

If you would like, I would be pleased to be involved in the effort to develop the Accessible Authentication SC.


John

John Rochford
UMass Medical School
Eunice Kennedy Shriver Center
Director, INDEX Program
Faculty, Family Medicine & Community Health
www.DisabilityInfo.org<http://<br/>www.DisabilityInfo.org>

________________________________
From: Alastair Campbell <acampbell@nomensa.com>
Sent: Thursday, July 12, 2018 8:21:35 AM
To: Jeanne Spellman; w3c-wai-gl@w3.org
Subject: Re: Measurable vs testable [was: Silver Requirements Issues]

Hi Jeanne,

A couple of things I’d like to contribute/clarify:

> JS: The test example we have been discussing is the COGA proposal for accessible authentication that was not included in 2.1.

The main issues there were more around feasibility rather than testability, the more difficult ones were things like Plain Language in navigation, as the context of the site has a big impact, and testability is less clear than most interface level things. Usability testing wouldn’t be the right tool for the job there, you’d need huge sample sizes.


> For example, Silver could define a usability test procedure -- based on standard UX techniques -- that any developer, QA tester, or a11y expert could perform themselves (they would not have to have a outside PwD test it) to determine if a login page conformed to Silver for accessible authentication.

I am very sceptical that this approach would provide a good ‘answer’ to the question. Even apart from the ease with which testing can be biased (and in this case there is good motivation to bias the results intentionally!), what would constitute a pass? 3 out of 6 people? It’s never 100%. There are fundamental issues with using usability testing in a direct way.

Also, what if there is a pattern out there which has been usability tested by others already, with published results, do you still need to test yours?


> While we have discussed using process as a means of measurement, that has not been our focus and probably would only apply to the higher levels of Silver -- if we include it at all.  There are issues with using process to determine accessibility and we don't have solutions for them yet.  We are eager to get ideas from the people with experience with using process to evaluate accessibility.

That’s fair (I’d say the same about usability testing), but I’d like to include it as an avenue to explore in the requirements.

I was going to create a PR but I don’t have write access on the repo (yet).

My suggested additions were:

<h4>Conformance Model</h4>

+ <p>There are several areas for exploration in how conformance can work. These opportunities may or may not be incorporated. Then need to work together, and that interplay will be governed by the design principles.</p>

Add a bullet under flexibility:

+ <li><strong>Process guidance</strong>: Some requirements may be more important for certain websites and be very dependent on context. For example using plain language terminology in navigation is very difficult to test reliably given the constraints and context an ecommerce site has compared to a public sector website. Providing for a process to follow and document may be more appropriate for some requirements than either measurable or task-based approaches.</li>

And on tech-neutral:

<h3>Technology Neutral</h3>

+ <p>The guidelines should cover all web technologies available to users. It is likely that a layer of the guidance will be written to be technology neutral, but the guidelines should be able to include criteria that do not work across all technologies.</p>
<p class="ednote">Technology neutral



Cheers,

-Alastair