Measurable vs testable [was: Silver Requirements Issues]

Wilco is right, "measureable" means that you can come up with a number 
or score and that score can be used to determine whether or not 
something conforms to an individual piece of guidance (aka success 
criterion).  This means it can be used in a regulatory environment.  We 
are using "measureable" as a term to contrast with the WCAG 2.x 
"testable".  In the Silver context, "measurable" means "more inclusive 
tests than just a pass fail test".

An important underlying goal for Silver is be able to use Silver in a 
regulatory environment.   Silver is not doing away with conformance, we 
propose to make Silver conformance more inclusive than the current 
statements that can be evaluated as a pass/fail (aka success criteria).  
Keep in mind that AGWG has deferred a number of success criteria 
proposals -- especially from COGA -- as "will be addressed in Silver".  
AGWG has asked the Silver Task Force to solve these accessibility-needs 
problems, and a more flexible measurement system is one of the answers.

The Silver Design Sprint included lawyers, regulators, and policy 
specialists.  The prototypes we are currently working on are based on 
some of their ideas or were evaluated by them.  At the Design Sprint, we 
did usability testing of the prototypes, and the lawyers in particular 
were in great demand to evaluate and refine proposals.  We had one group 
that worked specifically on more diverse measurements that could be used 
in a regulatory environment.  If you want more information about it, I 
recommend reading the Silver Design Sprint Final Report. 
https://www.w3.org/community/silver/draft-final-report-of-silver/ 
Specifically, Table 5 worked on measurement prototypes. The Appendix has 
links to the photos of the detailed paper prototypes they developed and 
a short video of their final presentation.

The test example we have been discussing is the COGA proposal for 
accessible authentication that was not included in 2.1.  For example, 
Silver could define a usability test procedure -- based on standard UX 
techniques -- that any developer, QA tester, or a11y expert could 
perform themselves (they would not have to have a outside PwD test it) 
to determine if a login page conformed to Silver for accessible 
authentication.

As I learned at the Design Sprint, in the UX world, usability testing 
can be done by anyone following directions or script, while user testing 
brings in people outside the project (in our world, PwDs).  I think 
Alastair was assuming in his comment that we were planning to require 
user testing with people with disabilities. While I personally think we 
should include the opportunity for user testing for high levels of 
conformance by large organizations, we realize the limitations of 
testing with PwD for basic or mid-level conformance.  Our current 
discussion of conformance measurements is usability testing that anyone 
can perform.

Some of the other measurement possibilities that we discussed at the 
Design Sprint were rubrics, distance from a baseline, or grading 
scales.  The individual piece of guidance could assign the 
measurement(s) best suited for it.  Certainly, the early versions of 
Silver will lean heavily on the current guidance with pass/fail tests.  
But it opens the possibility of being able to include important needs of 
people with disabilities that cannot be included in WCAG 2.x because of 
the structural requirements of a success criterion.

While we have discussed using process as a means of measurement, that 
has not been our focus and probably would only apply to the higher 
levels of Silver -- if we include it at all.  There are issues with 
using process to determine accessibility and we don't have solutions for 
them yet.  We are eager to get ideas from the people with experience 
with using process to evaluate accessibility.

If you are interested in a high level overview of the conformance 
prototype being developed, this is the slide deck for a presentation I 
did a couple weeks ago with some of the conformance ideas. 
https://docs.google.com/presentation/d/1EJN_KTXD0NaGGpNrwAf3i3idA-k1A0QR6Ctbk0re3nE/edit#slide=id.g3ca1179f10_0_26

Plain Language:  I have noticed some confusion about Silver and plain 
language.  At this time, we are not discussing whether Silver should 
require plain language use for a web site.  We are focusing on the 
structure of Silver, not the content of Silver. We are saying that 
Silver itself should be written in plain language.  We are currently 
running an experiment where people with expertise with plain language 
are "translating" four existing WCAG 2.x success criteria to plain 
language to show what is possible in presenting accessibility guidance.  
When we have all the results in, we will show this to  AGWG, probably in 
2-3 weeks.

Please feel free to contribute your ideas and prototypes of what Silver 
can look like.  We have a repo at https://github.com/w3c/silver.  You 
can see the two prototypes that we currently have at 
https://w3c.github.io/silver/prototypes/index.html  This summer, our 
plan is to build multiple versions of prototypes and perform user 
testing of the prototypes to refine them and determine the best 
direction for Silver.

We are actively looking for people to submit prototypes and ideas.  I 
realize that AGWG is currently focused on Techniques, and I don't want 
to distract people from that important work.  But if you have the time 
and want to write up an idea you have, please send it to the Silver list 
at public-silver@w3.org or submit a Github issue on the Silver repo.  If 
you are excited about the possibilities and want to work up a detailed 
prototype, please submit a pull request for the Silver repo at 
https://github.com/w3c/silver.

jeanne


On 7/10/2018 7:48 PM, Alastair Campbell wrote:
>
> On the measurability / process points, I think Jason covered one 
> method well.
>
> Allowing for some requirements to be scored rather than true/false may 
> make sense for some, the requirement for Silver should be to explore 
> that with an eye on how conformance and accessibility-supported will 
> work (or whatever concepts they become).
>
> I think it has mostly been me talking about using process, and I will 
> try to pull together a decent bit of text that could be used in the 
> requirements doc.
>
> As a quick overview, my thinking comes from the discussions about some 
> of the previously proposed COGA SCs, and what the right tool for the 
> job is.
>
> Some people have raised usability testing as potential way to pass 
> something, which I don’t think would work, even for organisations that 
> can afford it. Useful previous post:
>
> https://lists.w3.org/Archives/Public/w3c-wai-gl/2017AprJun/0654.html
>
> There are some ISO standards (e.g. 9001, 27001) which I have 
> second-hand knowledge of, but my understanding is that you have to do 
> an evaluation of something (e.g. the security risk factors) and 
> document your mitigations. It isn’t a concrete pass/fail, but you have 
> to justify your decisions. There are some strong default practices as 
> well, e.g. all our docs have version history at the front.
>
> So in an accessibility context the requirements for plain-language 
> might involve a process rather than an outcome.
>
> The guidance could start with when you need to consider it. E.g. Does 
> the navigation have over 20 items? Do you run usability testing on 
> your site already? (Other positive/negative factors).
>
> If your site is in scope (rather than your page perhaps?), then you 
> will need to document how you mitigated the issue, which could be by 
> using a dictionary of common words, running usability testing with 
> particular audiences, or another method.
>
> In that way there could be a **layer** of guidance that does not apply 
> to all websites, and encourages good, user-centred-design practice 
> when appropriate.
>
> If anyone is more familiar with this approach please chip in, I’m 
> trying to get more details for my own understanding. (Grrr, paywalls 
> on standards, what’s that about?!)
>
> Cheers,
>
> -Alastair
>
> *From:*White, Jason J
>
> A second element of measurability, as I understand the proposed 
> Requirements, is to move away from a simple tripartite pass/fail/not 
> applicable distinction. How this would work in the over-all 
> conformance scheme would need to be determined, of course.
>
> For example, if numerical scores are associated with various 
> accessibility requirements, then a conformance level could be defined 
> in terms of achieving certain scores in individual requirements, or in 
> aggregate across a set of requirements, or both. The first case 
> effectively reconstructs a pass/fail arrangement, but the second is 
> more complex (e.g., the average, or total score across four different 
> requirements must exceed a threshold in order for web content to 
> conform at a given level). This raises the question of what the 
> permissible scores would mean in various cases, all of which “pass” so 
> far as over-all conformance is concerned.
>
> I’m also definitely interested in process-based requirements as a 
> complement to content-based ones.
>
> ------------------------------------------------------------------------