Re: Sampling for Testability

Hi Wayne,

I feel like I’m picking up a thought from about three steps in, was this triggered by the Ability to Override SC? The one which says to test by changing the colours to black & white?

If so, as context for everyone else, the current SC text for Ability to override is:
No loss of content or functionality on a webpage is caused by overriding:
1.   font family to Verdana, or
2.   foreground and background to white on black, or
3.   line-height of all text to 1.5, letter-spacing to 0.1em, and word-spacing to (TBC).

This moves away from the “mechanism is available” approach, and the idea is that if you test with that baseline then most user-adaptations should work.

A user might use Comic Sans instead of Verdana, but any negative effects from changing font-family should be highlighted with a standard (readable) font like Verdana.

There is a bookmarklet you can add from here to see the effect: https://github.com/w3c/wcag21/issues/78#issuecomment-277743607


The issue with the color aspect is that a white text on a black *image* background will disappear, and you’ll see that in testing. However, if it is the other way around then that would not come up in testing and could cause issues for users. (I.e. black text on white background, and the user switches to a dark-on-light colour scheme).

Can anyone think of a 3rd way? (Not mechanism-available, not defining testing criteria, but something else.)

Cheers,

-Alastair


From: Wayne Dick <wayneedick@gmail.com>
Date: Monday, 6 February 2017 at 17:16
To: WCAG <w3c-wai-gl@w3.org>
Subject: Sampling for Testability
Resent-From: WCAG <w3c-wai-gl@w3.org>
Resent-Date: Monday, 6 February 2017 at 17:17

I think our old means of testing may be the problem faced by many new SCs. If a range of values is 16M we cannot look at all of them. How ever if we only pick 2 that may not be enough for a quirky page we cannot even predict, because we are all reasonable coders. Why not consider new methods of testing. Like sampling.
Take color. We have a sample space of 16M squared. Of those only a subset is viable since contrast is needed. But it is still a very big number. I it would make sense to compile a list distribution of the number of foreground and background colors used by web sites. Step 1 or 2 standard deviations beyond the mean add 1 to that number and test that many color pairs chosen randomly so that color contrast is maintained.
For COGA we could employ user testing. Once again computing sample size is straight forward.
We would have to give up determinism for acceptable probability. Most product testing is done that way. They don't test every car of the assembly line for crash survival.
When we have ranges to large to test by hand or machine, or we have tests that require user testing we can use statistics. That will give us .95 assurance, and really, do we do better with accessibility testing.
Wayne