W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > January 2013

RE: Aim and impact of random sampling

From: Velleman, Eric <evelleman@bartimeus.nl>
Date: Thu, 24 Jan 2013 18:40:19 +0000
To: Aurélien Levy <aurelien.levy@temesis.com>, "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>
Message-ID: <3D063CE533923349B1B52F26312B0A466E248400@s107ma.bart.local>
Hi all,

In my opinion there was another good argument in the call that we should consider: A random sample (even if it is small) can act as a simple sort of verification indicator of the results found with the structured sample. In that case, a few web pages would then be sufficient and add to the reasonable confidence of the results of the evaluation. Not sure if this needs to be optional or very academic.
Kindest regards,


Van: Aurélien Levy [aurelien.levy@temesis.com]
Verzonden: donderdag 24 januari 2013 17:36
Aan: public-wai-evaltf@w3.org
Onderwerp: Re: Aim and impact of random sampling

+1 that the sense of the comment I made on the survey I think this need
to be an option

> The assumption has been that an additional random sample will make
> sure that a tester's intitial sampling of pages has not left out pages
> that may expose problems no present in the intitial sample.
> That aim in itself is laudable, but for this to work, the sampling
> would need to be
> 1. independent of individual tester choices (i.e., automatic) -
>    which would need a definition, inside the methodology, of a
>    valid approach for truly random sampling. No one has even hinted on
>    a reliable way to do that - I believe there is none.
>    A mere calculaton of sample size for a desired level of confidence
>    would need to be based to the total number of a site's pages *and*
>    page states - a number that will usually be unknown.
> 2. Fairly represent not just pages, but also page states.
>    But crawling a site to derive a collection of URLS for
>    random sampling is not doable since many states (and there URLs or
>    DOM states) only come about as a result of human input.
> I hope I am not coming across as a pest if I say again that in my
> opinion, we are shooting ourselves in the foot if we make random
> sampling a mandatory part of the WCAG-EM. Academics will be happy,
> practitioners working to a budget will just stay away from it.
> Detlev
Received on Thursday, 24 January 2013 18:42:12 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:23 UTC