Re: Measurable vs testable [was: Silver Requirements Issues]

One of the issues raised earlier in the discussion is whether there should be “process” requirements in addition to substantive “content” requirements.

Conformance to process requirements cannot be measured independently of the content creator simply by analyzing the “product” of their work; but it could be independently reviewed by auditors who are given access to the necessary documentary evidence by the content producer.

The W3C has not, so far as I know, offered normative “process” guidance in the past. I think this avenue is worth exploring, however, since some accessibility requirements that are not amenable to measurement in themselves may be captured by establishing requirements for the development process. The latter doesn’t serve as a guarantee, but as a strong indicator suitable as a complementary set of conformance criteria.

From: Jeanne Spellman <jspellman@spellmanconsulting.com> on behalf of Jeanne Spellman <jspellman@spellmanconsulting.com>
Date: Monday, July 16, 2018 at 12:59
To: David MacDonald <david100@sympatico.ca>
Cc: "w3c-wai-gl@w3.org" <w3c-wai-gl@w3.org>
Subject: Re: Measurable vs testable [was: Silver Requirements Issues]
Resent-From: <w3c-wai-gl@w3.org>
Resent-Date: Monday, July 16, 2018 at 12:59


Thanks, David!  This document is very helpful, and isn't a document that I remember seeing before.  I have been working off an AGWG wiki page and the Silver designation of the 2.1 Gitgub Issues, but this is a much more useful form factor.  Bookmarked.  :)

We are NOT writing content yet, and I am relying on the various task forces to recommend 1 or 2 examples that they think are representative for Silver prototypes and user testing.  However, the notes are very helpful in identifying examples that would be useful for Silver the TFs may not have considered.

Thank you very much!

jeanne

On 7/16/2018 12:47 PM, David MacDonald wrote:
Here are a list of SCs that didn't make it into WCAG.

http://tinyurl.com/jmo9st4<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftinyurl.com%2Fjmo9st4&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C1%7C636673571894557336&sdata=GmuNyKx%2F4T1svcREumBueFi7QODItUs%2FzywuTyGe6Rw%3D&reserved=0>

They are found on the tab
"SCs Not accepted for 2.1"


Cheers,
David MacDonald



CanAdapt Solutions Inc.

Tel:  613.235.4902

LinkedIn
<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdavidmacdonald100&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894557336&sdata=H0b0wrCNcYJJUBYhmF%2Fdft0Qgc30w7JeHyXdmmgh1Cc%3D&reserved=0>

twitter.com/davidmacd<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftwitter.com%2Fdavidmacd&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894567344&sdata=0wUOmMLDNvcGTJKqtiH4Zyx6ufU2a9abWpU%2FXipo5ug%3D&reserved=0>

GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDavidMacDonald&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894577352&sdata=IWapQKrHWk3%2B%2BimJ4JC6YCpANX%2BjtCf69Vrki8xP3sE%3D&reserved=0>

www.Can-Adapt.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.can-adapt.com%2F&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894577352&sdata=P4nsVAQ3UCAdhmYCytnhkVIJr2QMe0%2FRIYCmwKWGrVI%3D&reserved=0>



  Adapting the web to all users
            Including those with disabilities

If you are not the intended recipient, please review our privacy policy<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.davidmacd.com%2Fdisclaimer.html&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894587361&sdata=b7EY3%2FsNWDyrKhDQtWwfMHxbgEc7qaH3m83ZGypcYYY%3D&reserved=0>

On Thu, Jul 12, 2018 at 9:49 AM, Rochford, John <john.rochford@umassmed.edu<mailto:john.rochford@umassmed.edu>> wrote:
Hi Jeanne,
I am the COGA Task Force manager for the Accessible Authentication SC. I am also the author of the Web Security and Privacy issue paper it is based on. See: https://rawgit.com/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frawgit.com%2Fw3c%2Fcoga%2Fmaster%2Fissue-papers%2Fprivacy-security.html&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894587361&sdata=VoivNZcUaSIGNkuUgzN9xaWYIhbc6NOniNys6Uth%2FF4%3D&reserved=0>w3c<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frawgit.com%2Fw3c%2Fcoga%2Fmaster%2Fissue-papers%2Fprivacy-security.html&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894597369&sdata=4vijkwRsB3iVxj2mvKu1dF7uQqz7sMMcPtLCVTIUEp0%3D&reserved=0>/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frawgit.com%2Fw3c%2Fcoga%2Fmaster%2Fissue-papers%2Fprivacy-security.html&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894607373&sdata=aZK6YaqjT6SnH%2FFmeAC%2FHTmO3NxxtZHADY9bclX%2BS8M%3D&reserved=0>coga<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frawgit.com%2Fw3c%2Fcoga%2Fmaster%2Fissue-papers%2Fprivacy-security.html&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894607373&sdata=aZK6YaqjT6SnH%2FFmeAC%2FHTmO3NxxtZHADY9bclX%2BS8M%3D&reserved=0>/master/issue-papers/privacy-security.html<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frawgit.com%2Fw3c%2Fcoga%2Fmaster%2Fissue-papers%2Fprivacy-security.html&data=02%7C01%7Cjjwhite%40ets.org%7Cf1f49d7e003e4c61294608d5eb3d8990%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C636673571894617382&sdata=npxTlt19oQkaHKhbe5%2FzLtKNqdosVUq5C3pyXh4D7vc%3D&reserved=0>
Accessible Authentication was the only COGA SC considered at Level A for 2.1. That it did not make it was due in part to the testability issues you and Alastair are discussing. However, the push back involved several other objections.
If you would like, I would be pleased to be involved in the effort to develop the Accessible Authentication SC.

John
John Rochford
UMass Medical School
Eunice Kennedy Shriver Center
Director, INDEX Program
Faculty, Family Medicine & Community Health
www.DisabilityInfo.org<http://%3Cbr/%3Ewww.DisabilityInfo.org>

________________________________
From: Alastair Campbell <acampbell@nomensa.com<mailto:acampbell@nomensa.com>>
Sent: Thursday, July 12, 2018 8:21:35 AM
To: Jeanne Spellman; w3c-wai-gl@w3.org<mailto:w3c-wai-gl@w3.org>
Subject: Re: Measurable vs testable [was: Silver Requirements Issues]

Hi Jeanne,

A couple of things I’d like to contribute/clarify:

> JS: The test example we have been discussing is the COGA proposal for accessible authentication that was not included in 2.1.

The main issues there were more around feasibility rather than testability, the more difficult ones were things like Plain Language in navigation, as the context of the site has a big impact, and testability is less clear than most interface level things. Usability testing wouldn’t be the right tool for the job there, you’d need huge sample sizes.


> For example, Silver could define a usability test procedure -- based on standard UX techniques -- that any developer, QA tester, or a11y expert could perform themselves (they would not have to have a outside PwD test it) to determine if a login page conformed to Silver for accessible authentication.
I am very sceptical that this approach would provide a good ‘answer’ to the question. Even apart from the ease with which testing can be biased (and in this case there is good motivation to bias the results intentionally!), what would constitute a pass? 3 out of 6 people? It’s never 100%. There are fundamental issues with using usability testing in a direct way.

Also, what if there is a pattern out there which has been usability tested by others already, with published results, do you still need to test yours?


> While we have discussed using process as a means of measurement, that has not been our focus and probably would only apply to the higher levels of Silver -- if we include it at all.  There are issues with using process to determine accessibility and we don't have solutions for them yet.  We are eager to get ideas from the people with experience with using process to evaluate accessibility.

That’s fair (I’d say the same about usability testing), but I’d like to include it as an avenue to explore in the requirements.

I was going to create a PR but I don’t have write access on the repo (yet).

My suggested additions were:

<h4>Conformance Model</h4>

+ <p>There are several areas for exploration in how conformance can work. These opportunities may or may not be incorporated. Then need to work together, and that interplay will be governed by the design principles.</p>

Add a bullet under flexibility:

+ <li><strong>Process guidance</strong>: Some requirements may be more important for certain websites and be very dependent on context. For example using plain language terminology in navigation is very difficult to test reliably given the constraints and context an ecommerce site has compared to a public sector website. Providing for a process to follow and document may be more appropriate for some requirements than either measurable or task-based approaches.</li>

And on tech-neutral:

<h3>Technology Neutral</h3>

+ <p>The guidelines should cover all web technologies available to users. It is likely that a layer of the guidance will be written to be technology neutral, but the guidelines should be able to include criteria that do not work across all technologies.</p>
<p class="ednote">Technology neutral



Cheers,

-Alastair






________________________________

This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.


Thank you for your compliance.

________________________________