RE: Size of random sample

Hi Detlev,

There are many ways to select a random sample and I am not sure if we want to put one or more of them into the methodology as it would seem we have a preference. In another thread I propose to add something to the reporting section: "describe how you selected the random sample." Probably evaluators will have this lying around somewhere and probably they use the same method to choose a random sample everytime they do an evaluation.

I very much like your step 3, 4 and 5 where you provide the opportunity to use the random sampling to add pages to the structured sample and also to filter the pages (now in a seperate section). 

"Stop when you have selected n additional pages (or the number of pages selected amounts to i% of you total sample, whichever condition is met earlier)"

What is 'n' and 'i' in your opinion :-)

Hope to hear what others think of this solution for size of the random sample and the possibility to use random sampling to add pages to the structured sample that have been missed before.
Kindest regards,

Eric




________________________________________
Van: Detlev Fischer [detlev.fischer@testkreis.de]
Verzonden: zondag 27 januari 2013 10:58
Aan: Velleman, Eric
CC: public-wai-evaltf@w3.org
Onderwerp: Re: Size of random sample

Hi all,

To clarify my position:

IF we find a way of describing an practical approach for random sampling (i.e. an approach that does not require a science degree and/or sophisticated tools), and make sure it can be applied across the very different evaluation cases WCAG-EM must deal with. I have no problem with making it mandatory. In my view, the description:

> "Include a random sample of at least 20 percent of the number of web pages that are in the structured sample or a minimum of 5 random web pages from the scope of the website into the sample (if available)."

does not meet those criteria of 'practical' and 'applicable' - it does not say anything about how you should do it and what will count as 'random'. A tester can do as she  or he likes and is not required to document how he or she arrived at the random sample (which is fine in my view, but may have little to do with a random procedure, and there is no way to check this independently).

An alternative more practical description in line with Richard's approach might be:

1. Conduct a site-specific search with an search engine (e.g., google site:www.testsite.com)
2. Pick every tenth (or nth, to be determined) entry on your hit list.
3. View each page and check if the type is already covered by your sample so far. If so, move on.
3. If you find a template not yet included or content types (such as video, data tables, forms,
   interactive content) not already covered, add the page to your sample
4. Stop when you have selected n additional pages (or the number of pages selected amounts to
   i% of you total sample, whichever condition is met earlier)

This will not always work and it will ignore dynamically generated pages that are not brought up by the crawler, but at least the tester knows what to do. The statement may be scoped to those cases where it does work (small to medium sized websites at are not primarily applications).

Regards,
Detlev


On 27 Jan 2013, at 01:12, Velleman, Eric wrote:

> Dear all,
>
> More discussion about the new random sample section:
> <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e>
> <https://www.w3.org/2002/09/wbs/48225/evaltfq7/results>
>
> 3. The size of the sample must depend upon the size and complexity of the site (number of pages and variety of content).
>
> In the structured part we ask people to include at least one page for every…
> For the random sample I tried to do the same. And at the same time trying to steer away from having to describe how to measure the size or complexity of a website and what this means for the size of the random sample.
> There is more research necessary and coming up, but for the moment we should put something here that we can used for testing. What about the following direction:
>
> Proposed resolution: "Include a random sample of at least 20 percent of the number of web pages that are in the structured sample or a minimum of 5 random web pages from the scope of the website into the sample (if available)."
>
> The '20' and the '5' are of course open for discussion. Please insert your own numbers.
> Please let me know if this is generally the direction we should explore further.
> Kindest regards,
>
> Eric
>

--
Detlev Fischer
testkreis - das Accessibility-Team von feld.wald.wiese
c/o feld.wald.wiese
Thedestraße 2
22767 Hamburg

Tel   +49 (0)40 439 10 68-3
Mobil +49 (0)1577 170 73 84
Fax   +49 (0)40 439 10 68-5

http://www.testkreis.de
Beratung, Tests und Schulungen für barrierefreie Websites

Received on Sunday, 27 January 2013 14:35:00 UTC