Comments on random sampling (Step 3.e)

Dear Eval TF,

References:
  - http://www.w3.org/TR/WCAG-EM/#step3e
  - http://www.w3.org/WAI/ER/conformance/comments-20130226#samplesize

Summary of comments:
  - Niel King: random sampling is not necessary.
  - Sylvie Duchateau: 5% (of website) is too much; needs a cap.
  - Loic Martinez Normand: please explain why 5% (or 5 pages).
  - Sharron Rush: 5% (of the structured sample) is too little.

Note: Sylvie missed the point that the size of the random sample is 5% 
of the size of the structured sample (or 5 pages), and not 5% of the 
size of the website. We might need to further clarify our wording here.

Loic asks for additional clarification and suggests that the random 
sample is more linked to verification of results than to increasing 
confidence in the results. We can discuss this but it seems to be a 
somewhat different type of comment and discussion then the rest here.

We remain with Niel who argues that random sampling is not necessary, 
while Sharron argues that the random sample size should be increased.

We have previously discussed and agreed that random sampling is useful 
for verification of the results (which is why we added this section); 
there does not seem to be new information or reasons to remove it. The 
question is if we need to adjust the size of the random sample.

Questions for the group:
  - What were the typical sizes of the samples during the test runs?
  - How well did the random sample work as a verification indicator?
  - Other observations from doing evaluations outside the test runs?

Regards,
   Shadi

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Tuesday, 2 July 2013 19:05:40 UTC