RE: To sample, or not to sample

Hi Richard, all,

I feel that sampling is almost always necessary. Only if the website is very small (less than 25 p), you can skip sampling.

I think the sampling process is very valuable. The main reason: it saves time. Using a sample to check a website's accessibility saves a lot of time compared to evaluating every page. My evaluations are mostly manual, if I have to evaluate more than 25 pages the added confidence in the result is seldom worth the added time spent on the evaluation.

I like the way we have it now: If not-sampling is not an option due to some constraint, you can make a sample. As most websites are so big that evaluating every page takes to much time, I think it would be counterproductive to stress the need for full analysis more than we do now. If people have the impression they always have to evaluate every page of a website, they might shun away from using this methodology.

Because most websites will not be fully evaluated, I don't think we need to specify different situations in which a sample is needed. We can add some examples of situations in which you would use a sample, but I don't think this is very necessary.

1),2),3) In our current version, we base the size of the sample on the variation in content:

"Depending on the size and complexity of the website, the size of this sample will vary. For example, a website with few types of web pages that are all generated from a confined set of templates, such as a data repository, will likely require a smaller sample than a website with many types of web pages that are authored using different mechanisms. Also a web application could have a limited number of web pages that dynamically generate content with varying types of appearance, behavior, and information that each need to be sampled."

I like this pragmatic approach. If we add the homogeneity/heterogeneity of a website as a third factor, besides size and complexity, I think the resulting samples will give a realistic view of the accessibility problems that may exist.

All the best,

Martijn Houtepen

Stichting Accessibility
030 - 239 8270





________________________________
Van: Userite [mailto:richard@userite.com]
Verzonden: donderdag 23 mei 2013 22:54
Aan: Eval TF
Onderwerp: To sample, or not to sample

In todays conference Eric proposed that I ask the following question in the list.

When is it better to skip the sampling process and simply evaluate the whole site?

Producing a reliable sample can be quite complex (judging by the amount of discussions we have had on this topic!) and can never compete 100% with a full audit for reliability and confidence. I therefore believe that our methodology should stress the need for a full analysis UNLESS this is not viable. And here we need to offer advice and guidance on how to decide when it is necessary to evaluate only a sample of pages rather than the whole site.   We need some method to help the user to decide:-
1) What level of confidence is required  when auditing a particular site (for example is it a high profile site? or one that is likely to have a high level of disabled users?)
2) How to judge the difference in effort required to do a full audit immediately, or to conduct a sampling method first and then identify, justify and finally audit the resulting sections.
3) How to balance the results of 1) and 2) with the available budget.

I often find that quite reasonable sites (even up to 50 pages or so) are more efficiently evaluated if I check the entire site. I admit that this is a "gut reaction" rather than the result of scientific analysis.

Does anyone have any better ideas (i.e. not gut reaction) for how we might do this?


Regards
Richard


Richard Warren
Technical Manager
Website Auditing Limited (Userite)
http://www.website-accessibility.com



Richard Warren
Technical Manager
Website Auditing Limited (Userite)
http://www.website-accessibility.com