Re: All pages

Hi Kathy, All,

As I understand it, the comment is that WCAG-EM currently requires 
sampling in all cases. It does not seem to adequately address the 
situation where someone wants to evaluate all pages on a site (for 
whatever reasons).

The original comment is here:
  - 
<http://lists.w3.org/Archives/Public/public-wcag-em-comments/2012Apr/0004>

My suggestion is simply to acknowledge that evaluators can ignore the 
sampling bits in WCAG-EM for cases where they want to evaluate all 
pages. I think this is implied anyway since selecting all pages in a 
site includes any sample set selected according to WCAG-EM, but some 
careful wording (especially to emphasize that in many cases it is not 
economically feasible nor useful to evaluate all pages) may be a helpful 
clarification.

Again, I'm not really suggesting change but clarification that people 
are welcome to evaluate all pages if they really want to...

Best,
   Shadi


On 26.6.2012 14:38, Kathleen Wahlbin wrote:
> Hi -
>
>
>
> During our conversations early on, we talked about using a sampling method
> to review a set of pages on the site and then automated tools to check the
> full site.  Is the recommendation now to check all pages manually?
>
>
>
> If we are suggesting checking all pages manually, then I think we need to be
> careful about what this means for different types of websites/web
> applications.   Here are two situations (and I am sure there are many more
> that we could come up with):
>
>
>
> -        For applications, a page could have many different variations
> depending on the data or options selected.  Do all of these different
> variations need to be checked?
>
>
>
> -        For database driven websites, there may be a lot of different pages
> but they may all use the same template and the data or content of the page
> may be same.  In this case, if the content is added to the page in the same
> way, then an evaluator should be able to test just one of these pages rather
> than the full set.
>
>
>
> Kathy
>
>
>
>   -----Original Message-----
> From: Shadi Abou-Zahra [mailto:shadi@w3.org]
> Sent: Tuesday, June 26, 2012 6:21 AM
> To: Alistair Garrison
> Cc: Eval TF
> Subject: Re: All pages
>
>
>
> Hi Alistair,
>
>
>
> I'm not sure what you mean by overkill. Maybe it is not economically
> feasible to check all pages but ideally all pages are checked before making
> an accessibility statement about them (especially for small websites where
> the use of templates, content management systems, and other quality
> assurance procedures are often less sophisticated).
>
>
>
> We should bear in mind that no matter how robust a sampling method is, there
> is always a possibility that an evaluator misses critical parts of the
> website through sampling. Sampling is always an approximation but ideally it
> is "close enough" to reality except for few edge cases.
>
>
>
> What is the problem with saying something like "if you can check all the
> pages then please go ahead and ignore the sampling section"?
>
>
>
> Sidenote: A production line already assumes large amounts of products that
> are produced in the same way, so that sampling becomes effective for quality
> assurance. However, not all websites are developed this way, and most
> websites actually resemble a handicraft store... ;)
>
>
>
> Regards,
>
>     Shadi
>
>
>
>
>
> On 26.6.2012 11:56, Alistair Garrison wrote:
>
>> Hi Shadi, Richard,
>
>>
>
>> Re-reading your emails you seem to be looking at conformance evaluation
> from a new viewpoint (maybe rightly / maybe wrongly?) from the one we seem
> to have adopted to date.
>
>>
>
>> Viewpoint 1 (to date) is an evaluator wishing to see if a whole website
> conforms with WCAG 2.0.
>
>>
>
>> With regard to this viewpoint an evaluator doesn't need to check all
> pages, simply enough to confirm whether the website conforms or doesn't.  If
> there are several examples of content which pass a checkpoint and the same
> team has built all the site, I suppose you can assume that the other
> relevant instances of content will also be ok, and visa versa for content
> with issues.  You would not think about checking all items on a production
> line, would you?
>
>> Historically most people seem to think this type of evaluation is
> efficiently and effectively achieved by sampling, which is reflected in
> their different methodologies.  Checking all pages could be seen as
> unnecessary 'overkill' even on small sites...
>
>>
>
>> Viewpoint 2 (new) is a website owner wanting to find and correct all
> faults in their website - in which case they would want to look at all pages
> no matter what the size.
>
>>
>
>> Either way, treating the concept as editorial only sounds to me a bit
> casual, but I'd be interested to hear other views...
>
>>
>
>> All the best
>
>>
>
>> Alistair
>
>>
>
>> On 26 Jun 2012, at 11:12, Shadi Abou-Zahra wrote:
>
>>
>
>>> I agree with this approach too. The default (and ideal) would be to check
> all pages. In cases where this is not practically feasible we provide a
> robust sampling procedure.
>
>>>
>
>>> This probably affects several sections, including the introduction,
> though rather editorially only.
>
>>>
>
>>> Regards,
>
>>>    Shadi
>
>>>
>
>>>
>
>>> On 25.6.2012 19:47, RichardWarren wrote:
>
>>>> Michael,
>
>>>> We are not suggesting "all or nothing" .
>
>>>> We are saying that the preferred method is to validate all pages,
>
>>>> but if this is too large a task (which for typically large sites it
>
>>>> will be) then here is a sampling procedure which will ensure that
>
>>>> all important elements are covered.
>
>>>>
>
>>>> Thus owners of small sites that want to check their compliance can
>
>>>> skip the sampling process and get straight on with the method of
>
>>>> validating their site.
>
>>>>
>
>>>> Richard
>
>>>>
>
>>>> -----Original Message----- From: Michael S Elledge
>
>>>> Sent: Monday, June 25, 2012 3:34 PM
>
>>>> To:  <mailto:public-wai-evaltf@w3.org> public-wai-evaltf@w3.org
>
>>>> Cc: Alistair Garrison ; RichardWarren ; Eval TF
>
>>>> Subject: Re: All pages
>
>>>>
>
>>>> Hi All--
>
>>>>
>
>>>> I agree with Alistair. We nearly always test a sample of pages in a
>
>>>> website. Although it would be ideal to test every page in a site, it
>
>>>> is impractical because of time and cost, especially if it is
>
>>>> performed manually. Many people reading our methodology will be
>
>>>> looking to apply it to their reviews, which out of necessity will be
> based on sampling.
>
>>>> The alternative, relying solely on automated checkers to review a
>
>>>> medium to large site in its entirety, I think we can all agree is
>
>>>> not a viable alternative, even with their improvements.
>
>>>>
>
>>>> We spent a significant amount of time describing sampling approaches
>
>>>> early in this process, so I'm surprised that the "all or nothing"
>
>>>> approach is still being debated. I may have missed something along
>
>>>> the way, however, so please forgive me if I did.
>
>>>>
>
>>>> Best regards,
>
>>>>
>
>>>> Mike
>
>>>>
>
>>>> On 6/25/2012 3:08 AM, Alistair Garrison wrote:
>
>>>>> Hi Richard,
>
>>>>>
>
>>>>> Reading the archive I see we have talked around the subject of
>
>>>>> sampling - but not actually whether to evaluate all pages instead
>
>>>>> of a sample. Reading a number of emails, however, it becomes clear
>
>>>>> that we all seem to use some kind of sampling effort - hence its
>
>>>>> seemingly automatic acceptance to this point.
>
>>>>>
>
>>>>> To my mind, there are many reasons for adopting our reasonably
>
>>>>> straight-forward sample-based approach (again we have all mostly
>
>>>>> done something similar for years), even for smaller sites, over
>
>>>>> evaluating all pages.  I suppose its lower cost in terms of time /
>
>>>>> effort - with the same actual benefits is one of the top reasons for
> sampling.
>
>>>>>
>
>>>>> I'm also worried that the changes you suggest (did it also need a
>
>>>>> change to the Requirements docs) at this stage will create a
>
>>>>> two-tier (all or sample) approach, forking our current work and
>
>>>>> possibly opening a big can of worms (like how do you realistically,
>
>>>>> and with very high confidence, find all pages in a website, what
>
>>>>> exactly is a small or medium site, etc...).
>
>>>>>
>
>>>>> I remain to be convinced, but I would be interested to hear the
>
>>>>> views of others.
>
>>>>>
>
>>>>> All the best
>
>>>>>
>
>>>>> Alistair
>
>>>>>
>
>>>>> On 22 Jun 2012, at 12:05, RichardWarren wrote:
>
>>>>>
>
>>>>>> Reason for making the default position to include all pages
>
>>>>>> (entire
>
>>>>>> website)
>
>>>>>>
>
>>>>>> 1) Taking the Internet (WWW) as a whole, the majority of sites are
>
>>>>>> quite small (100 or so pages), typically things like "Mum&  Pop"
>
>>>>>> stores, SME profiles, personal or project websites.
>
>>>>>>
>
>>>>>> 2) Where this is practical a full evaluation is more reliable than
>
>>>>>> a sample.
>
>>>>>>
>
>>>>>> 3) Our brief is to deliver an evaluation methodology, not a
>
>>>>>> sampling methodology.
>
>>>>>>
>
>>>>>> 4) Reliable sampling is a complex procedure, if owners of
>
>>>>>> small/medium sites think they have to go through sampling they
>
>>>>>> will give up.
>
>>>>>>
>
>>>>>> 5) Sampling procedure will only be required for large sites so it
>
>>>>>> should be an option. The default should be to evaluate the whole
>
>>>>>> site. If the evaluator feels that is too large a task then s/he
>
>>>>>> should have the option to use a sampling procedure to help manage
>
>>>>>> the evaluation work load.
>
>>>>>>
>
>>>>>> My feeling as that we need to change the order of our text so that
>
>>>>>> sampling is offered as the option, not the full audit.
>
>>>>>>
>
>>>>>> Richard
>
>>>>>>
>
>>>>>> -----Original Message----- From: Alistair Garrison
>
>>>>>> Sent: Friday, June 22, 2012 10:39 AM
>
>>>>>> To: RichardWarren ; Eval TF
>
>>>>>> Subject: All pages
>
>>>>>>
>
>>>>>> Hi Richard,
>
>>>>>>
>
>>>>>> We were not able to debate the agenda item relating to "testing
>
>>>>>> all pages"? Can you just remind me what was behind this issue?
>
>>>>>>
>
>>>>>> All the best
>
>>>>>>
>
>>>>>> Alistair
>
>>>>>>
>
>>>>>
>
>>>>>
>
>>>>
>
>>>
>
>>> --
>
>>> Shadi Abou-Zahra -  <http://www.w3.org/People/shadi/>
> http://www.w3.org/People/shadi/ Activity Lead,
>
>>> W3C/WAI International Program Office Evaluation and Repair Tools
>
>>> Working Group (ERT WG) Research and Development Working Group (RDWG)
>
>>>
>
>>>
>
>>>
>
>>
>
>>
>
>
>
> --
>
> Shadi Abou-Zahra -  <http://www.w3.org/People/shadi/>
> http://www.w3.org/People/shadi/ Activity Lead, W3C/WAI International Program
> Office Evaluation and Repair Tools Working Group (ERT WG) Research and
> Development Working Group (RDWG)
>
>
>
>
>
>
>
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Tuesday, 26 June 2012 13:11:41 UTC