Re: Measurable vs testable [was: Silver Requirements Issues]

  John,  thank you for your offer!  :)

All:

We are not yet working on the Silver *content*.  I want to be very clear 
about that to everyone.  Right now, we are working on the *structure *of 
Silver.  However, we do need concrete examples of needs of people with 
disabilities that AGWG has deferred to Silver.   We need to be able to 
test whether the Silver design can incorporate that guidance.

Silver is currently building prototypes of how the Silver structure 
could work.  We want to make it easy for people to submit prototypes for 
the structure, by giving them sample text they can use in their 
prototypes.  That way, the people working on prototypes don't have to 
either make up their own text, or (shudder) use lorem ipsum text in 
their prototypes.  :)

At this point, the most helpful thing John Rochford could contribute 
would be some:

  * general guidance text on Accessible Authentication (in plain language)
  * tests for Accessible Authentication

*This is **not a content proposal*, it is just an example that Silver 
Community Group and Task Force members can use while we are building 
prototypes.  That way, we can load the example into the prototypes.  If 
someone from COGA could also write some general text and tests for the 
Plain Language in Navigation example that Alastair recommended Silver 
look at, that would also be helpful.  I think a lot of this text already 
exists, and just needs to be pulled together.

We have also asked for examples from Low Vision TF and Mobile TF of 
success criteria that were deferred to Silver that the Task Force would 
like us to consider for evaluating the Silver structure to determine if 
the accessibility needs that they identified can be included in Silver.  
If any AGWG members know of some other representative examples of WCAG 
2.1 proposals that were deferred to Silver and would like to write up 
some sample text we could use for testing, that would be very helpful.

We are building multiple prototypes of potential Silver structure so we 
can compare, refine, and evaluate them with user testing. That is our 
main project for this summer.  We want to bring a refined prototype to 
TPAC with user testing result data.  It's a big project, and we 
appreciate any contributions from individual members of AGWG.  (I don't 
want to distract or hijack the main AGWG work.) Silver prototypes are 
asynchronous work, and anyone who wants to contribute can do so.  You do 
not have to attend Silver conference calls (which are mostly coordination).

Thanks again for your offer, John!  It helps Silver to have the experts 
in a specific accessibility need write up the example guidance text, so 
we have a clear and representative example to use in the prototypes.

If anyone wants to contribute example text and wants more details, 
please contact me off-list and I'll help you get started.

jeanne


On 7/12/2018 9:49 AM, Rochford, John wrote:
> Hi Jeanne,
>
> I am the COGA Task Force manager for the Accessible Authentication SC. 
> I am also the author of the Web Security and Privacy issue paper it is 
> based on. See: https://rawgit.com/ 
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>w3c 
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/ 
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>coga 
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>/master/issue-papers/privacy-security.html 
> <https://rawgit.com/w3c/coga/master/issue-papers/privacy-security.html>
>
> Accessible Authentication was the only COGA SC considered at Level A 
> for 2.1. That it did not make it was due in part to the testability 
> issues you and Alastair are discussing. However, the push back 
> involved several other objections.
>
> If you would like, I would be pleased to be involved in the effort to 
> develop the Accessible Authentication SC.
>
>
> John
>
> John Rochford
> UMass Medical School
> Eunice Kennedy Shriver Center
> Director, INDEX Program
> Faculty, Family Medicine & Community Health
> www.DisabilityInfo.org <http://%3Cbr/%3Ewww.DisabilityInfo.org>
>
> ------------------------------------------------------------------------
> *From:* Alastair Campbell <acampbell@nomensa.com>
> *Sent:* Thursday, July 12, 2018 8:21:35 AM
> *To:* Jeanne Spellman; w3c-wai-gl@w3.org
> *Subject:* Re: Measurable vs testable [was: Silver Requirements Issues]
>
> Hi Jeanne,
>
> A couple of things I’d like to contribute/clarify:
>
> > JS: The test example we have been discussing is the COGA proposal 
> for accessible authentication that was not included in 2.1.
>
> The main issues there were more around feasibility rather than 
> testability, the more difficult ones were things like Plain Language 
> in navigation, as the context of the site has a big impact, and 
> testability is less clear than most interface level things. Usability 
> testing wouldn’t be the right tool for the job there, you’d need huge 
> sample sizes.
>
> > For example, Silver could define a usability test procedure -- based 
> on standard UX techniques -- that any developer, QA tester, or a11y 
> expert could perform themselves (they would not have to have a outside 
> PwD test it) to determine if a login page conformed to Silver for 
> accessible authentication.
>
> I am very sceptical that this approach would provide a good ‘answer’ 
> to the question. Even apart from the ease with which testing can be 
> biased (and in this case there is good motivation to bias the results 
> intentionally!), what would constitute a pass? 3 out of 6 people? It’s 
> never 100%. There are fundamental issues with using usability testing 
> in a direct way.
>
> Also, what if there is a pattern out there which has been usability 
> tested by others already, with published results, do you still need to 
> test yours?
>
> > While we have discussed using process as a means of measurement, 
> that has not been our focus and probably would only apply to the 
> higher levels of Silver -- if we include it at all.  There are issues 
> with using process to determine accessibility and we don't have 
> solutions for them yet.  We are eager to get ideas from the people 
> with experience with using process to evaluate accessibility.
>
> That’s fair (I’d say the same about usability testing), but I’d like 
> to include it as an avenue to explore in the requirements.
>
> I was going to create a PR but I don’t have write access on the repo 
> (yet).
>
> My suggested additions were:
>
> <h4>Conformance Model</h4>
>
> + <p>There are several areas for exploration in how conformance can 
> work. These opportunities may or may not be incorporated. Then need to 
> work together, and that interplay will be governed by the design 
> principles.</p>
>
> Add a bullet under flexibility:
>
> + <li><strong>Process guidance</strong>: Some requirements may be more 
> important for certain websites and be very dependent on context. For 
> example using plain language terminology in navigation is very 
> difficult to test reliably given the constraints and context an 
> ecommerce site has compared to a public sector website. Providing for 
> a process to follow and document may be more appropriate for some 
> requirements than either measurable or task-based approaches.</li>
>
> And on tech-neutral:
>
> <h3>Technology Neutral</h3>
>
> + <p>The guidelines should cover all web technologies available to 
> users. It is likely that a layer of the guidance will be written to be 
> technology neutral, but the guidelines should be able to include 
> criteria that do not work across all technologies.</p>
> <p class="ednote">Technology neutral
>
> Cheers,
>
> -Alastair
>