Re: Measurable vs testable [was: Silver Requirements Issues]

Thanks, David!  This document is very helpful, and isn't a document that 
I remember seeing before.  I have been working off an AGWG wiki page and 
the Silver designation of the 2.1 Gitgub Issues, but this is a much more 
useful form factor.  Bookmarked. :)

We are NOT writing content yet, and I am relying on the various task 
forces to recommend 1 or 2 examples that they think are representative 
for Silver prototypes and user testing.  However, the notes are very 
helpful in identifying examples that would be useful for Silver the TFs 
may not have considered.

Thank you very much!

jeanne


On 7/16/2018 12:47 PM, David MacDonald wrote:
> Here are a list of SCs that didn't make it into WCAG.
>
> http://tinyurl.com/jmo9st4
>
> They are found on the tab
> "SCs Not accepted for 2.1"
>
> Cheers,
> David MacDonald
>
> *Can**Adapt**Solutions Inc.*
>
> Tel:  613.235.4902
>
> LinkedIn
> <http://www.linkedin.com/in/davidmacdonald100>
>
> twitter.com/davidmacd <http://twitter.com/davidmacd>
>
> GitHub <https://github.com/DavidMacDonald>
>
> www.Can-Adapt.com <http://www.can-adapt.com/>
>
> /  Adapting the web to *all* users/
>
> /            Including those with disabilities/
>
> If you are not the intended recipient, please review our privacy 
> policy <http://www.davidmacd.com/disclaimer.html>
>
> On Thu, Jul 12, 2018 at 9:49 AM, Rochford, John 
> <john.rochford@umassmed.edu <mailto:john.rochford@umassmed.edu>> wrote:
>
>     Hi Jeanne,
>
>     I am the COGA Task Force manager for the Accessible Authentication
>     SC. I am also the author of the Web Security and Privacy issue
>     paper it is based on. See: https://rawgit.com/
>     <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>w3c
>     <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/
>     <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>coga
>     <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/master/issue-papers/privacy-security.html
>     <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>
>
>     Accessible Authentication was the only COGA SC considered at Level
>     A for 2.1. That it did not make it was due in part to the
>     testability issues you and Alastair are discussing. However, the
>     push back involved several other objections.
>
>     If you would like, I would be pleased to be involved in the effort
>     to develop the Accessible Authentication SC.
>
>
>     John
>
>     John Rochford
>     UMass Medical School
>     Eunice Kennedy Shriver Center
>     Director, INDEX Program
>     Faculty, Family Medicine & Community Health
>     www.DisabilityInfo.org <http://%3Cbr/%3Ewww.DisabilityInfo.org>
>
>     ------------------------------------------------------------------------
>     *From:* Alastair Campbell <acampbell@nomensa.com
>     <mailto:acampbell@nomensa.com>>
>     *Sent:* Thursday, July 12, 2018 8:21:35 AM
>     *To:* Jeanne Spellman; w3c-wai-gl@w3.org <mailto:w3c-wai-gl@w3.org>
>     *Subject:* Re: Measurable vs testable [was: Silver Requirements
>     Issues]
>
>     Hi Jeanne,
>
>     A couple of things I’d like to contribute/clarify:
>
>     > JS: The test example we have been discussing is the COGA
>     proposal for accessible authentication that was not included in 2.1.
>
>     The main issues there were more around feasibility rather than
>     testability, the more difficult ones were things like Plain
>     Language in navigation, as the context of the site has a big
>     impact, and testability is less clear than most interface level
>     things. Usability testing wouldn’t be the right tool for the job
>     there, you’d need huge sample sizes.
>
>     > For example, Silver could define a usability test procedure --
>     based on standard UX techniques -- that any developer, QA tester,
>     or a11y expert could perform themselves (they would not have to
>     have a outside PwD test it) to determine if a login page conformed
>     to Silver for accessible authentication.
>
>     I am very sceptical that this approach would provide a good
>     ‘answer’ to the question. Even apart from the ease with which
>     testing can be biased (and in this case there is good motivation
>     to bias the results intentionally!), what would constitute a pass?
>     3 out of 6 people? It’s never 100%. There are fundamental issues
>     with using usability testing in a direct way.
>
>     Also, what if there is a pattern out there which has been
>     usability tested by others already, with published results, do you
>     still need to test yours?
>
>     > While we have discussed using process as a means of measurement,
>     that has not been our focus and probably would only apply to the
>     higher levels of Silver -- if we include it at all.  There are
>     issues with using process to determine accessibility and we don't
>     have solutions for them yet.  We are eager to get ideas from the
>     people with experience with using process to evaluate accessibility.
>
>     That’s fair (I’d say the same about usability testing), but I’d
>     like to include it as an avenue to explore in the requirements.
>
>     I was going to create a PR but I don’t have write access on the
>     repo (yet).
>
>     My suggested additions were:
>
>     <h4>Conformance Model</h4>
>
>     + <p>There are several areas for exploration in how conformance
>     can work. These opportunities may or may not be incorporated. Then
>     need to work together, and that interplay will be governed by the
>     design principles.</p>
>
>     Add a bullet under flexibility:
>
>     + <li><strong>Process guidance</strong>: Some requirements may be
>     more important for certain websites and be very dependent on
>     context. For example using plain language terminology in
>     navigation is very difficult to test reliably given the
>     constraints and context an ecommerce site has compared to a public
>     sector website. Providing for a process to follow and document may
>     be more appropriate for some requirements than either measurable
>     or task-based approaches.</li>
>
>     And on tech-neutral:
>
>     <h3>Technology Neutral</h3>
>
>     + <p>The guidelines should cover all web technologies available to
>     users. It is likely that a layer of the guidance will be written
>     to be technology neutral, but the guidelines should be able to
>     include criteria that do not work across all technologies.</p>
>     <p class="ednote">Technology neutral
>
>     Cheers,
>
>     -Alastair
>
>