Re: Sampling for Testability

Hi Alastair et. al.,

This is overall. In 2.0 we were very deterministic. That is a good start,
but testable is broader than deterministic testing. Most products are
tested by sampling, why not web sites.

In low vision we usually want a range so representative sampling is in
order. In COGA we may need to test with users.

This was not in 2.0. That's probably because we are programmers and don't
write programs that work 95% of the time. Testing could be .95, and that is
considered good.

We have a year and a half before this has to be released. We can work on
test methodology. That was all I was saying.

Hope your day went well.

Wayne


On Mon, Feb 6, 2017 at 10:04 AM, Alastair Campbell <acampbell@nomensa.com>
wrote:

> Hi Wayne,
>
>
>
> I feel like I’m picking up a thought from about three steps in, was this
> triggered by the Ability to Override SC? The one which says to test by
> changing the colours to black & white?
>
>
>
> If so, as context for everyone else, the current SC text for Ability to
> override is:
>
> No loss of content or functionality on a webpage is caused by overriding:
>
> 1.   font family to Verdana, or
>
> 2.   foreground and background to white on black, or
>
> 3.   line-height of all text to 1.5, letter-spacing to 0.1em, and
> word-spacing to (TBC).
>
>
>
> This moves away from the “mechanism is available” approach, and the idea
> is that if you test with that baseline then most user-adaptations should
> work.
>
>
>
> A user might use Comic Sans instead of Verdana, but any negative effects
> from changing font-family should be highlighted with a standard (readable)
> font like Verdana.
>
>
>
> There is a bookmarklet you can add from here to see the effect:
> https://github.com/w3c/wcag21/issues/78#issuecomment-277743607
>
>
>
> The issue with the color aspect is that a white text on a black **image**
> background will disappear, and you’ll see that in testing. However, if it
> is the other way around then that would not come up in testing and could
> cause issues for users. (I.e. black text on white background, and the user
> switches to a dark-on-light colour scheme).
>
>
>
> Can anyone think of a 3rd way? (Not mechanism-available, not defining
> testing criteria, but something else.)
>
>
>
> Cheers,
>
>
>
> -Alastair
>
>
>
>
>
> *From: *Wayne Dick <wayneedick@gmail.com>
> *Date: *Monday, 6 February 2017 at 17:16
> *To: *WCAG <w3c-wai-gl@w3.org>
> *Subject: *Sampling for Testability
> *Resent-From: *WCAG <w3c-wai-gl@w3.org>
> *Resent-Date: *Monday, 6 February 2017 at 17:17
>
>
>
> I think our old means of testing may be the problem faced by many new SCs.
> If a range of values is 16M we cannot look at all of them. How ever if we
> only pick 2 that may not be enough for a quirky page we cannot even
> predict, because we are all reasonable coders. Why not consider new methods
> of testing. Like sampling.
>
> Take color. We have a sample space of 16M squared. Of those only a subset
> is viable since contrast is needed. But it is still a very big number. I it
> would make sense to compile a list distribution of the number of foreground
> and background colors used by web sites. Step 1 or 2 standard deviations
> beyond the mean add 1 to that number and test that many color pairs chosen
> randomly so that color contrast is maintained.
>
> For COGA we could employ user testing. Once again computing sample size is
> straight forward.
>
> We would have to give up determinism for acceptable probability. Most
> product testing is done that way. They don't test every car of the assembly
> line for crash survival.
>
> When we have ranges to large to test by hand or machine, or we have tests
> that require user testing we can use statistics. That will give us .95
> assurance, and really, do we do better with accessibility testing.
>
> Wayne
>