AW: To sample, or not to sample

Hi Richard, I know you are fast due to long experience but looking at my own style of checking (and counting in what I know of my fellow testers) one hour per page on average is what we need in most cases (bar very simple sites). So testing 50 pages fully would be prohibitive in terms of cost/time needed.  It also usually would not reveal many problems beyond those already identified on a smaller, well selected sample. The confidence of 'having done the whole lot' would depend on the degree of thoroughness during checking. Both breadth and thoroughness cost time, and usually some trade-off needs to be made to be able to conduct an evaluation at a cost acceptable to the client. My point is that I think we cannot and should not define a threshold at which to recommend to test all pages instead of sampling. We all know that 'a page' can be everything, from a tiny step in some process to a full-timeblown application.
Best, Detlev
Sent from my phone

Userite <richard@userite.com> wrote:

>In todays conference Eric proposed that I ask the following question in the list.
>
> 
>
>When is it better to skip the sampling process and simply evaluate the whole site?
>
> 
>
>Producing a reliable sample can be quite complex (judging by the amount of discussions we have had on this topic!) and can never compete 100% with a full audit for reliability and confidence. I therefore believe that our methodology should stress the need for a full analysis UNLESS this is not viable. And here we need to offer advice and guidance on how to decide when it is necessary to evaluate only a sample of pages rather than the whole site.   We need some method to help the user to decide:-
>
>1) What level of confidence is required  when auditing a particular site (for example is it a high profile site? or one that is likely to have a high level of disabled users?)
>
>2) How to judge the difference in effort required to do a full audit immediately, or to conduct a sampling method first and then identify, justify and finally audit the resulting sections.
>
>3) How to balance the results of 1) and 2) with the available budget.
>
> 
>
>I often find that quite reasonable sites (even up to 50 pages or so) are more efficiently evaluated if I check the entire site. I admit that this is a “gut reaction” rather than the result of scientific analysis. 
>
> 
>
>Does anyone have any better ideas (i.e. not gut reaction) for how we might do this?
>
> 
>
> 
>
>Regards
>
>Richard
>
> 
>
> 
>
>Richard Warren
>Technical Manager
>Website Auditing Limited (Userite)
>http://www.website-accessibility.com
>
> 
>
> 
>
> 
>
>Richard Warren
>Technical Manager
>Website Auditing Limited (Userite)
>http://www.website-accessibility.com
>