W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > January 2013

RE: Size of random sample

From: Vivienne CONWAY <v.conway@ecu.edu.au>
Date: Thu, 31 Jan 2013 21:46:32 +0800
To: Detlev Fischer <detlev.fischer@testkreis.de>, "Velleman, Eric" <evelleman@bartimeus.nl>
CC: "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>
Message-ID: <8AFA77741B11DB47B24131F1E38227A901007237325D@XCHG-MS1.ads.ecu.edu.au>
Hi Detlev
I like this method - it makes a lot of sense.  Thanks.


Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.
From: Detlev Fischer [detlev.fischer@testkreis.de]
Sent: Sunday, 27 January 2013 6:58 PM
To: Velleman, Eric
Cc: public-wai-evaltf@w3.org
Subject: Re: Size of random sample

Hi all,

To clarify my position:

IF we find a way of describing an practical approach for random sampling (i.e. an approach that does not require a science degree and/or sophisticated tools), and make sure it can be applied across the very different evaluation cases WCAG-EM must deal with. I have no problem with making it mandatory. In my view, the description:

> "Include a random sample of at least 20 percent of the number of web pages that are in the structured sample or a minimum of 5 random web pages from the scope of the website into the sample (if available)."

does not meet those criteria of 'practical' and 'applicable' - it does not say anything about how you should do it and what will count as 'random'. A tester can do as she  or he likes and is not required to document how he or she arrived at the random sample (which is fine in my view, but may have little to do with a random procedure, and there is no way to check this independently).

An alternative more practical description in line with Richard's approach might be:

1. Conduct a site-specific search with an search engine (e.g., google site:www.testsite.com)
2. Pick every tenth (or nth, to be determined) entry on your hit list.
3. View each page and check if the type is already covered by your sample so far. If so, move on.
3. If you find a template not yet included or content types (such as video, data tables, forms,
   interactive content) not already covered, add the page to your sample
4. Stop when you have selected n additional pages (or the number of pages selected amounts to
   i% of you total sample, whichever condition is met earlier)

This will not always work and it will ignore dynamically generated pages that are not brought up by the crawler, but at least the tester knows what to do. The statement may be scoped to those cases where it does work (small to medium sized websites at are not primarily applications).


On 27 Jan 2013, at 01:12, Velleman, Eric wrote:

> Dear all,
> More discussion about the new random sample section:
> <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e>
> <https://www.w3.org/2002/09/wbs/48225/evaltfq7/results>
> 3. The size of the sample must depend upon the size and complexity of the site (number of pages and variety of content).
> In the structured part we ask people to include at least one page for every…
> For the random sample I tried to do the same. And at the same time trying to steer away from having to describe how to measure the size or complexity of a website and what this means for the size of the random sample.
> There is more research necessary and coming up, but for the moment we should put something here that we can used for testing. What about the following direction:
> Proposed resolution: "Include a random sample of at least 20 percent of the number of web pages that are in the structured sample or a minimum of 5 random web pages from the scope of the website into the sample (if available)."
> The '20' and the '5' are of course open for discussion. Please insert your own numbers.
> Please let me know if this is generally the direction we should explore further.
> Kindest regards,
> Eric

Detlev Fischer
testkreis - das Accessibility-Team von feld.wald.wiese
c/o feld.wald.wiese
Thedestraße 2
22767 Hamburg

Tel   +49 (0)40 439 10 68-3
Mobil +49 (0)1577 170 73 84
Fax   +49 (0)40 439 10 68-5

Beratung, Tests und Schulungen für barrierefreie Websites

This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

Received on Thursday, 31 January 2013 13:47:26 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:23 UTC