Re: Measurable vs testable [was: Silver Requirements Issues]

Here are a list of SCs that didn't make it into WCAG.

http://tinyurl.com/jmo9st4

They are found on the tab
"SCs Not accepted for 2.1"

Cheers,
David MacDonald



*Can**Adapt* *Solutions Inc.*

Tel:  613.235.4902

LinkedIn
<http://www.linkedin.com/in/davidmacdonald100>

twitter.com/davidmacd

GitHub <https://github.com/DavidMacDonald>

www.Can-Adapt.com <http://www.can-adapt.com/>



*  Adapting the web to all users*
*            Including those with disabilities*

If you are not the intended recipient, please review our privacy policy
<http://www.davidmacd.com/disclaimer.html>

On Thu, Jul 12, 2018 at 9:49 AM, Rochford, John <john.rochford@umassmed.edu>
wrote:

> Hi Jeanne,
>
> I am the COGA Task Force manager for the Accessible Authentication SC. I
> am also the author of the Web Security and Privacy issue paper it is based
> on. See: https://rawgit.com/
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>w3c
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>
> coga
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/
> master/issue-papers/privacy-security.html
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>
>
> Accessible Authentication was the only COGA SC considered at Level A for
> 2.1. That it did not make it was due in part to the testability issues you
> and Alastair are discussing. However, the push back involved several other
> objections.
>
> If you would like, I would be pleased to be involved in the effort to
> develop the Accessible Authentication SC.
>
>
> John
>
> John Rochford
> UMass Medical School
> Eunice Kennedy Shriver Center
> Director, INDEX Program
> Faculty, Family Medicine & Community Health
> www.DisabilityInfo.org <http://%3Cbr/%3Ewww.DisabilityInfo.org>
>
> ------------------------------
> *From:* Alastair Campbell <acampbell@nomensa.com>
> *Sent:* Thursday, July 12, 2018 8:21:35 AM
> *To:* Jeanne Spellman; w3c-wai-gl@w3.org
> *Subject:* Re: Measurable vs testable [was: Silver Requirements Issues]
>
>
> Hi Jeanne,
>
>
>
> A couple of things I’d like to contribute/clarify:
>
>
>
> > JS: The test example we have been discussing is the COGA proposal for
> accessible authentication that was not included in 2.1.
>
>
>
> The main issues there were more around feasibility rather than
> testability, the more difficult ones were things like Plain Language in
> navigation, as the context of the site has a big impact, and testability is
> less clear than most interface level things. Usability testing wouldn’t be
> the right tool for the job there, you’d need huge sample sizes.
>
>
>
>
>
> > For example, Silver could define a usability test procedure -- based on
> standard UX techniques -- that any developer, QA tester, or a11y expert
> could perform themselves (they would not have to have a outside PwD test
> it) to determine if a login page conformed to Silver for accessible
> authentication.
>
> I am very sceptical that this approach would provide a good ‘answer’ to
> the question. Even apart from the ease with which testing can be biased
> (and in this case there is good motivation to bias the results
> intentionally!), what would constitute a pass? 3 out of 6 people? It’s
> never 100%. There are fundamental issues with using usability testing in a
> direct way.
>
>
>
> Also, what if there is a pattern out there which has been usability tested
> by others already, with published results, do you still need to test yours?
>
>
>
> > While we have discussed using process as a means of measurement, that
> has not been our focus and probably would only apply to the higher levels
> of Silver -- if we include it at all.  There are issues with using process
> to determine accessibility and we don't have solutions for them yet.  We
> are eager to get ideas from the people with experience with using process
> to evaluate accessibility.
>
> That’s fair (I’d say the same about usability testing), but I’d like to
> include it as an avenue to explore in the requirements.
>
> I was going to create a PR but I don’t have write access on the repo (yet).
>
> My suggested additions were:
>
> <h4>Conformance Model</h4>
>
> + <p>There are several areas for exploration in how conformance can work.
> These opportunities may or may not be incorporated. Then need to work
> together, and that interplay will be governed by the design principles.</p>
>
> Add a bullet under flexibility:
>
> + <li><strong>Process guidance</strong>: Some requirements may be more
> important for certain websites and be very dependent on context. For
> example using plain language terminology in navigation is very difficult to
> test reliably given the constraints and context an ecommerce site has
> compared to a public sector website. Providing for a process to follow and
> document may be more appropriate for some requirements than either
> measurable or task-based approaches.</li>
>
> And on tech-neutral:
>
> <h3>Technology Neutral</h3>
>
> + <p>The guidelines should cover all web technologies available to users.
> It is likely that a layer of the guidance will be written to be technology
> neutral, but the guidelines should be able to include criteria that do not
> work across all technologies.</p>
> <p class="ednote">Technology neutral
>
>
>
> Cheers,
>
> -Alastair
>
>
>