RE: testing with a "High degree of confidence"

David,

I don't exactly recall all the reasons, but these are probably most of it.

First, WCAG 2.0 covers many disabilities.  Even if 10 human subjects is statistically adequate (see the third reason), you need 10 times an undetermined number of disabilities to cover WCAG 2.0.

Second, it is hard to establish the knowledge threshold of your human subjects.  Do you get 10 experts, complete novices, a mix, or just random human subjects?  Add to the mix the degree of specialty of the site matters.  Does the human subject have to be financially literate to test a stock trading site, for example?  How do you measure the degree of financially literacy in correspondence to the task?  This issue is a challenge in terms design of experiment.  The difference in result would be dramatic depending on how the experiment was designed.

Third, 10 is simply inadequate by any statistical measure.  Just a couple of "off-target" human subjects will throw your analysis way off course.  A sample size of a 100 is the bare minimum by rule of thumb.

Lastly, 80% is plugged out of the air, so to speak.  (I don't remember if we talked about 8 out of 10, but I'm using your number per this mail.) Why shouldn't it be 75% or 84.37%?  There is nothing scientifically significant about the number 80%.

Bottom line is that such approach is at best anecdote and certainly not scientific.  The degree of confidence of the approach would be unacceptably low.  In general, conducting analysis with a small sample size is more suitable for qualitative analyses like focus groups and the like, which generally does not give you a pass/fail result.  (BTW, 10 is still too small for most focus groups.) Hope that helps.

All best,
Alex

From: David MacDonald [mailto:david100@sympatico.ca]
Sent: Tuesday, April 02, 2013 9:11 AM
To: ryladog@earthlink.net; 'Loretta Guarino Reid'; 'Michael Cooper'; 'Gregg Vanderheiden'
Cc: 'WCAG WG'; 'Eval TF'
Subject: RE: testing with a "High degree of confidence"

Thanks Katie

Can you remember where the vestiges of it ended up in the WCAG documents....if at all?
I'm just looking to see if we require a high correlation among experts... or simply high level of confidence...perhaps they are not necessarily the same thing.

Cheers
David MacDonald

CanAdapt Solutions Inc.
  Adapting the web to all users
            Including those with disabilities
www.Can-Adapt.com<http://www.can-adapt.com/>

From: Katie Haritos-Shea EARTHLINK [mailto:ryladog@earthlink.net]
Sent: April-02-13 11:11 AM
To: 'David MacDonald'; 'Loretta Guarino Reid'; 'Michael Cooper'; 'Gregg Vanderheiden'
Cc: 'WCAG WG'; 'Eval TF'
Subject: RE: testing with a "High degree of confidence"

David,

I remember these discussions back when - I recall Gregg providing the 8 out of 10  - and - I brought his up for WCAG Evaluation Methodology working group - for their puposes they wanted an algorithmic reference - not human judgment.   Am not sure they found one.

Katie

From: David MacDonald [mailto:david100@sympatico.ca]
Sent: Monday, April 01, 2013 3:50 PM
To: Loretta Guarino Reid; Michael Cooper; Gregg Vanderheiden
Cc: WCAG WG; Eval TF
Subject: testing with a "High degree of confidence"

I remember early drafts of WCAG, when discussing human testing we said it was dependable human testing if "8 of 10 testers would come to the same conclusions... " or something like that...we later changed it to something like "most testers would come to the same conclusions" because we thought the 8 out of 10 rule was a bit prescriptive.

I've been looking for that in the WCAG 2, or the Understanding conformance, Understanding WCAG etc... and didn't find it.

The closest I could find was this, but it seems to be more related to automatic testing...
Does anyone remember the history of that line about "most experts woukd agree..." and where it is now?

"The Success Criteria can be tested by a combination of machine and human evaluation as long as it is possible to determine whether a Success Criterion has been satisfied with a high level of confidence."

http://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html#uc-accessibility-support-head

Cheers
David MacDonald

CanAdapt Solutions Inc.
  Adapting the web to all users
            Including those with disabilities
www.Can-Adapt.com<http://www.can-adapt.com/>

No virus found in this message.
Checked by AVG - www.avg.com<http://www.avg.com>
Version: 2013.0.3267 / Virus Database: 3161/6218 - Release Date: 04/01/13