To sample, or not to sample

In todays conference Eric proposed that I ask the following question in the list.

When is it better to skip the sampling process and simply evaluate the whole site?

Producing a reliable sample can be quite complex (judging by the amount of discussions we have had on this topic!) and can never compete 100% with a full audit for reliability and confidence. I therefore believe that our methodology should stress the need for a full analysis UNLESS this is not viable. And here we need to offer advice and guidance on how to decide when it is necessary to evaluate only a sample of pages rather than the whole site.   We need some method to help the user to decide:-
1) What level of confidence is required  when auditing a particular site (for example is it a high profile site? or one that is likely to have a high level of disabled users?)
2) How to judge the difference in effort required to do a full audit immediately, or to conduct a sampling method first and then identify, justify and finally audit the resulting sections.
3) How to balance the results of 1) and 2) with the available budget.

I often find that quite reasonable sites (even up to 50 pages or so) are more efficiently evaluated if I check the entire site. I admit that this is a “gut reaction” rather than the result of scientific analysis. 

Does anyone have any better ideas (i.e. not gut reaction) for how we might do this?


Regards
Richard


Richard Warren
Technical Manager
Website Auditing Limited (Userite)
http://www.website-accessibility.com



Richard Warren
Technical Manager
Website Auditing Limited (Userite)
http://www.website-accessibility.com