Re: Sampling for Testability

Sampling is how WCAG is tested on large sites ... and this is in the WCAG
Testing Methodology document.

https://www.w3.org/TR/WCAG-EM/

I would suggest WCAG has never been deterministic. Otherwise we wouldn't
have come up with the term high "inter rater reliability" .



Cheers,
David MacDonald



*Can**Adapt* *Solutions Inc.*

Tel:  613.235.4902

LinkedIn
<http://www.linkedin.com/in/davidmacdonald100>

twitter.com/davidmacd

GitHub <https://github.com/DavidMacDonald>

www.Can-Adapt.com <http://www.can-adapt.com/>



*  Adapting the web to all users*
*            Including those with disabilities*

If you are not the intended recipient, please review our privacy policy
<http://www.davidmacd.com/disclaimer.html>

On Mon, Feb 6, 2017 at 2:14 PM, Wayne Dick <wayneedick@gmail.com> wrote:

> Hi Alastair et. al.,
>
> This is overall. In 2.0 we were very deterministic. That is a good start,
> but testable is broader than deterministic testing. Most products are
> tested by sampling, why not web sites.
>
> In low vision we usually want a range so representative sampling is in
> order. In COGA we may need to test with users.
>
> This was not in 2.0. That's probably because we are programmers and don't
> write programs that work 95% of the time. Testing could be .95, and that is
> considered good.
>
> We have a year and a half before this has to be released. We can work on
> test methodology. That was all I was saying.
>
> Hope your day went well.
>
> Wayne
>
>
> On Mon, Feb 6, 2017 at 10:04 AM, Alastair Campbell <acampbell@nomensa.com>
> wrote:
>
>> Hi Wayne,
>>
>>
>>
>> I feel like I’m picking up a thought from about three steps in, was this
>> triggered by the Ability to Override SC? The one which says to test by
>> changing the colours to black & white?
>>
>>
>>
>> If so, as context for everyone else, the current SC text for Ability to
>> override is:
>>
>> No loss of content or functionality on a webpage is caused by overriding:
>>
>> 1.   font family to Verdana, or
>>
>> 2.   foreground and background to white on black, or
>>
>> 3.   line-height of all text to 1.5, letter-spacing to 0.1em, and
>> word-spacing to (TBC).
>>
>>
>>
>> This moves away from the “mechanism is available” approach, and the idea
>> is that if you test with that baseline then most user-adaptations should
>> work.
>>
>>
>>
>> A user might use Comic Sans instead of Verdana, but any negative effects
>> from changing font-family should be highlighted with a standard (readable)
>> font like Verdana.
>>
>>
>>
>> There is a bookmarklet you can add from here to see the effect:
>> https://github.com/w3c/wcag21/issues/78#issuecomment-277743607
>>
>>
>>
>> The issue with the color aspect is that a white text on a black **image**
>> background will disappear, and you’ll see that in testing. However, if it
>> is the other way around then that would not come up in testing and could
>> cause issues for users. (I.e. black text on white background, and the user
>> switches to a dark-on-light colour scheme).
>>
>>
>>
>> Can anyone think of a 3rd way? (Not mechanism-available, not defining
>> testing criteria, but something else.)
>>
>>
>>
>> Cheers,
>>
>>
>>
>> -Alastair
>>
>>
>>
>>
>>
>> *From: *Wayne Dick <wayneedick@gmail.com>
>> *Date: *Monday, 6 February 2017 at 17:16
>> *To: *WCAG <w3c-wai-gl@w3.org>
>> *Subject: *Sampling for Testability
>> *Resent-From: *WCAG <w3c-wai-gl@w3.org>
>> *Resent-Date: *Monday, 6 February 2017 at 17:17
>>
>>
>>
>> I think our old means of testing may be the problem faced by many new
>> SCs. If a range of values is 16M we cannot look at all of them. How ever if
>> we only pick 2 that may not be enough for a quirky page we cannot even
>> predict, because we are all reasonable coders. Why not consider new methods
>> of testing. Like sampling.
>>
>> Take color. We have a sample space of 16M squared. Of those only a subset
>> is viable since contrast is needed. But it is still a very big number. I it
>> would make sense to compile a list distribution of the number of foreground
>> and background colors used by web sites. Step 1 or 2 standard deviations
>> beyond the mean add 1 to that number and test that many color pairs chosen
>> randomly so that color contrast is maintained.
>>
>> For COGA we could employ user testing. Once again computing sample size
>> is straight forward.
>>
>> We would have to give up determinism for acceptable probability. Most
>> product testing is done that way. They don't test every car of the assembly
>> line for crash survival.
>>
>> When we have ranges to large to test by hand or machine, or we have tests
>> that require user testing we can use statistics. That will give us .95
>> assurance, and really, do we do better with accessibility testing.
>>
>> Wayne
>>
>
>