- From: Shadi Abou-Zahra <shadi@w3.org>
- Date: Tue, 31 Jan 2012 13:31:35 +0100
- To: Léonie Watson <lwatson@nomensa.com>
- CC: Eval TF <public-wai-evaltf@w3.org>
Yes, very good point. I think what to do when failures are encountered is probably outside the scope of the Methodology but a small paragraph on sampling for re-testing certainly makes sense. Best, Shadi On 31.1.2012 12:30, Léonie Watson wrote: > Imagine a scenario where a sample of pages has been tested. The results show several failed criteria. The website owner goes away to fix the failed criteria, before the sample is retested for conformance. > > My suggestion is that after the website owner has fixed the failed criteria, it isn't only the original sample of pages that is tested. Instead it's a combination of pages from the original sample and randomly selected new pages. > > The idea being that if the website owner had been able to apply the fixes to the new sample of pages, as well as the original sample, it would be a fair indication that the fixes would also be available across the rest of the website as well. > > As I said, this doesn't get us closer to absolute conformance across 100% of the website, but it could act as an indicator of the likelihood that the rest of the site met the same conformance level as the original sample. > > Hope I've explained this better. > > Léonie. > > > > > > > -- > Nomensa - humanising technology > > Léonie Watson, Director of Accessibility& Web Development > tel: +44 (0)117 929 7333 > mob: +44 (0)792 116 8551 > twitter: @we_are_Nomensa @LeonieWatson > > Nomensa Email Disclaimer: http://www.nomensa.com/email-disclaimer > > -----Original Message----- > From: Shadi Abou-Zahra [mailto:shadi@w3.org] > Sent: 31 January 2012 07:15 > To: Léonie Watson > Cc: public-wai-evaltf@w3.org > Subject: Re: [summary attempt] 100% conformance for the pages sampled... > > Hi Leonie, > > I'm not entirely sure I understand what you mean by re-testing. > > I think there is general agreement that we are talking about websites that are too large to allow evaluation of every single page. The idea is that a carefully selected sample of web pages could provide a good representation of the website as a whole. > > It is clear that evaluating such a representative sample is always an approximation, and that there is no 100% certainty. A critical aspect of our work is to develop a good definition for representative sample that maximizes confidence in the accuracy of this approximation while remaining at a manageable size (= cost). > > An important point that was made is that if the sample does not conform to WCAG2 then the website as a whole also doesn't. The Methodology will need to reflect this aspect and also provide guidance for recording the failures found in the sample. Recording the failures consistently is an important aspect in the procurement process (for example to compare the > bids) and to help guide web developers. > > Best, > Shadi > > > On 30.1.2012 15:46, Léonie Watson wrote: >> Thanks Shadi. That's a useful summary. It seems we have a gap between the quantity of pages that can realistically be audited and the quantity that would have to be covered before a complete conformance statement could be issued. >> >> I can't immediately think of a way to audit an entire site. Not the big ones at least. It seems to me that our best bet might be to try and reduce the gap in some way? >> >> In part this comes down to the website owner's ability to take the results from the initial sample and apply the solutions to the rest of the site. This can be measured (to some extent) by retesting half the original sample of pages, and the same number of new pages from elsewhere in the site. >> >> This wouldn't provide a 100% certainty of site wide conformance of course. It could provide an indication of how likely the possibility of conformance might be though. >> >> >> Léonie. >> >> >> -- >> Nomensa - humanising technology >> >> Léonie Watson, Director of Accessibility& Web Development >> tel: +44 (0)117 929 7333 >> mob: +44 (0)792 116 8551 >> twitter: @we_are_Nomensa @LeonieWatson >> >> Nomensa Email Disclaimer: http://www.nomensa.com/email-disclaimer >> >> -----Original Message----- >> From: Shadi Abou-Zahra [mailto:shadi@w3.org] >> Sent: 30 January 2012 14:18 >> To: public-wai-evaltf@w3.org >> Subject: [summary attempt] 100% conformance for the pages sampled... >> >> Dear Eval TF, >> >> Thank you all for the lively discussion on the mailing list, please continue to share your opinions and experiences! >> >> This is an attempt to summarize this discussion so far to help make decisions but it is by no means an attempt to stop the discussions. >> >> The thread starts here: >> >> -<http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Jan/0082> >> >> >> # Summary >> [[ >> It seems that several people agree that because evaluating a sample only covers a relatively small portion of a website and because our mission is to determine conformance to WCAG 2.0 (rather than redefine WCAG 2.0 conformance), the entire sample to should conform to the [target level of] WCAG 2.0. >> >> Some people have argued that many sited would then not meet WCAG 2.0 conformance, in particular with respect to incidental errors that can be commonly found on websites. Some counter arguments included that website owners can be given a chance to fix such incidental errors as part of an evaluation process (but outside the scope of this document) and that remaining errors should be documented to aide comparison of websites/products (such as in a procurement setting). How to document the encountered errors can be part of the reporting section. >> ]] >> >> >> Please feel free to make additions, clarifications, or remarks on this! >> >> Best, >> Shadi >> >> >> On 29.1.2012 11:12, Kerstin Probiesch wrote: >>> Hi Eric, all, >>> >>>> Thanks. I would like to avoid the word violations, but we could >>>> change the title to something like margin for failures or failure tolerance. >>>> Please feel free to propose a better term. >>> >>> What about "expected degree of conformance". In the text we can say: >>> no degrees, 100% for the sample. >>> >>>> We have agreed that the methodology will have a 0% margin for failures. >>>> This seems more strict than is normal in scientific research where >>>> mostly an error margin of 5 percent is accepted for the sample. >>> >>> When we just have a look on failures. As written in another mail we >>> should be careful with terms. Error margin is very close to "Margin of error". >>> Wikipedia gives a short definition: "The margin of error is a >>> statistic expressing the amount of random sampling error in a >>> survey's results. The larger the margin of error, the less faith one >>> should have that the poll's reported results are close to the "true" >>> figures; that is, the figures for the whole population. Margin of >>> error occurs whenever a population is incompletely sampled." Or >>> >>> Margin of error occurs whenever a website is incompletely sampled. >>> >>>> Agree that we also need to describe the possibility of sampling >>>> errors like you propose (that would then be errors made in selecting >>>> the sample, by tools or manual). Shall we add it to the sampling section? >>> >>> I think, for the moment and we decide later upon this. >>> >>> I'm no master of statistics and it's a couple of years ago that I was >>> bothered with test theory, scaling and all this. But I'm a bit >>> concerned about our terminology. Shouldn't we collect resources with >>> understandable explanations/definitions of specific terms in a Wiki? >>> This might also be important for the glossary of our methodology, >>> especially if we are using specific terms in another way (which we >>> should avoid in general) and also because of future translations. >>> >>> Best >>> >>> Kerstin >>> >>> >>>> Kindest regards, >>>> >>>> Eric >>>> >>>> >>>> ________________________________________ >>>> Van: Kerstin Probiesch [k.probiesch@googlemail.com] >>>> Verzonden: vrijdag 27 januari 2012 7:06 >>>> Aan: Velleman, Eric >>>> CC: Alistair Garrison; EvalTF >>>> Onderwerp: Re: 100% conformance for the pages sampled... >>>> >>>> Hi Eric, all >>>> >>>> I still don't agree with the section "Error margin". As pointed out: >>>> error margin a specific term. We should not use this term, when we >>>> speak about failures on pages. >>>> >>>> To make clear what we mean, we should use something like "failures" >>>> or "violations..." as heading for this section. Nevertheless we also >>>> need a section "error margin", where we describe possible systematic >>>> errors like sampling errors. I remember that we discussed also the >>>> term "systematic error" just in the context of failures on pages. As >>>> this an evaluation methodology I believe we should be careful with >>>> terms and shouldn't use specific terms of testing theories and >>>> methodologies in general for anything else to avoid confusion. >>>> >>>> Best >>>> >>>> Kerstin >>>> >>>> Am 27.01.2012 um 01:18 schrieb "Velleman, Eric" >>>> <evelleman@bartimeus.nl>: >>>> >>>>> Hi Alistair, >>>>> >>>>> The 100% conformance you refer to in your mail made it into the >>>> document in clause 5.5 Error Margin as we decided it: "The >>>> Methodology assumes a strict conformance requirement for all >>>> resources in the sample. I could change strict into 100%." >>>>> >>>>> Also in the same section I added a disclaimer following the outcome >>>> of our discussion. We said that it is necessary to say that we >>>> cannot guarantee that there will not be another error somewhere on the site. >>>> This is caused by the use of a sample: "It is important to note that >>>> due to the use of a sample there is never a 100% guarantee that the >>>> full website is conformant." >>>>> >>>>> I thought it might be good to explain that further and I propose to >>>> do that in the Random Resources clause (4.1.3). I already made a >>>> short >>>> link: "This specific error margin for a given confidence level is >>>> described in the section about Random Resources." We will discuss >>>> clause 4.1 next week (as agreed today), so we can then discuss if we >>>> want to keep that in the document or take it out again. >>>>> >>>>> Kindest regards, >>>>> >>>>> Eric >>>>> >>>>> >>>>> ________________________________ >>>>> Van: Alistair Garrison [alistair.j.garrison@gmail.com] >>>>> Verzonden: donderdag 26 januari 2012 15:50 >>>>> Aan: Eval TF >>>>> Onderwerp: Fwd: 100% conformance for the pages sampled... >>>>> >>>>> Dear All, >>>>> >>>>> The concept of "100% conformance for the pages sampled" gained a >>>> reasonable level of consensus amongst EVAL TF members (6 or 7 people). >>>> Several other ideas have seemingly gained the same group approval, >>>> but then fallen by the wayside as we've move ahead to debate >>>> something else... >>>>> >>>>> I'd like to know, what mechanism are we following to capture such >>>> ideas, and ensure that they are appear in 'black& white' in our >>>> methodology updates? >>>>> >>>>> All the best >>>>> >>>>> Alistair >>>>> >>>>> Begin forwarded message: >>>>> >>>>> From: Alistair Garrison >>>> <alistair.j.garrison@gmail.com<mailto:alistair.j.garrison@gmail.com> >>>>> >>>>> Date: 20 January 2012 14:10:20 CET >>>>> To: Léonie Watson<lwatson@nomensa.com<mailto:lwatson@nomensa.com>>, >>>> Eval TF<public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org>> >>>>> Subject: Re: 100% conformance for the pages sampled... >>>>> >>>>> Hi Léonie, >>>>> >>>>>> From http://www.w3.org/TR/WCAG/#conformance-claims - >>>>>> "Conformance is >>>> not possible at a particular level if any page in the process does >>>> not conform at that level or better". >>>>> >>>>> And, as an additional component of the conformance claim "A list of >>>> success criteria beyond the level of conformance claimed that have >>>> been met. This information should be provided in a form that users >>>> can use, preferably machine-readable metadata." >>>>> >>>>> So, I would suggest that for our purposes (those of evaluating the >>>> whole site) this would mean - making a level AA conformance claim >>>> for all pages in the sample (website). If required by the website >>>> owner (or other) it could be provided along with a list of urls for >>>> the web pages on which the additional AAA Success Criteria have been achieved. >>>>> >>>>> If we were not looking at the whole site separate AA and AAA claims >>>> for the relevant pages would probably be the way to go. I should >>>> add, that a AAA conformance claim could only be made for the pages >>>> which were AAA conformant - ruling out, in my opinion, a AAA with >>>> exceptions type claim. >>>>> >>>>> Let me know what you think. >>>>> >>>>> All the best >>>>> >>>>> Alistair >>>>> >>>>> On 20 Jan 2012, at 12:59, Léonie Watson wrote: >>>>> >>>>> Thanks Alastair, it does. Although it's started me thinking >>>>> more >>>> about this. Given that WCAG acknowledges that Level AAA is not >>>> possible with all technologies, we could end up with a sample that >>>> is partly Level AA compliant and partly Level AAA compliant. >>>>> >>>>> I'm thinking out loud here as much as anything, but would we >>>> therefore consider the sample to be 100% Level AA compliant with >>>> additional achievements, Level AAA compliant with exceptions, or >>>> part Level AA and part Level AAA as applicable? >>>>> >>>>> >>>>> Léonie. >>>>> >>>>> -- >>>>> Nomensa - humanising technology >>>>> >>>>> Léonie Watson, Director of Accessibility& Web Development >>>>> tel: +44 (0)117 929 7333 >>>>> mob: +44 (0)792 116 8551 >>>>> twitter: @we_are_Nomensa @LeonieWatson >>>>> >>>>> Nomensa Email Disclaimer: http://www.nomensa.com/email-disclaimer >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> From: Alistair Garrison [mailto:alistair.j.garrison@gmail.com] >>>>> Sent: 20 January 2012 09:38 >>>>> To: Eval TF >>>>> Subject: Re: 100% conformance for the pages sampled... >>>>> >>>>> Hi Leonie, >>>>> >>>>>> From http://www.w3.org/TR/WCAG/#conformance-claims - >>>>>> "Conformance is >>>> defined only for Web pages. However, a conformance claim may be made >>>> to cover one page, a series of pages, or multiple related Web pages." >>>>> >>>>> Where they say "for Web Pages" I think they meant to say "for full >>>> Web Pages" - referencing "Conformance (and conformance level) is for >>>> full Web page(s) only, and cannot be achieved if part of a Web page >>>> is excluded." from the same page. >>>>> >>>>> Hope this clarifies things... >>>>> >>>>> All the best >>>>> >>>>> Alistair >>>>> >>>>> On 20 Jan 2012, at 10:14, Léonie Watson wrote: >>>>> >>>>> If I understand correctly, WCAG conformance relates to a single >>>>> page, >>>> not to an entire website. I'd be interested in people's thoughts on >>>> how this might (or might not) influence our thinking about an error margin? >>>>> >>>>> >>>>> Léonie. >>>>> >>>>> -- >>>>> Nomensa - humanising technology >>>>> >>>>> Léonie Watson, Director of Accessibility& Web Development >>>>> tel: +44 (0)117 929 7333 >>>>> mob: +44 (0)792 116 8551 >>>>> twitter: @we_are_Nomensa @LeonieWatson >>>>> >>>>> Nomensa Email Disclaimer: http://www.nomensa.com/email-disclaimer >>>>> >>>>> -----Original Message----- >>>>> From: Detlev Fischer [mailto:fischer@dias.de] >>>>> Sent: 19 January 2012 21:58 >>>>> To: public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org> >>>>> Subject: Re: 100% conformance for the pages sampled... >>>>> >>>>> Let's stop here and consider the implications. >>>>> >>>>> Here and then, people in the EVAL TF have agreed that the 100% >>>> conformant site does not really exist 'out there'. Aren't we holding >>>> the bone a wee bit too high? I wonder what that will mean for the >>>> practical acceptance of the methodology. Will it come to be derided >>>> as academic, as impossibly demanding? Who then is the customer of a >>>> (sorry, chap) refused seal of conformance who bows to gracefully >>>> accept the list of flaws to rectify? Just wondering...it just >>>> strikes me as slightly surreal... >>>>> >>>>> Detlev >>>>> >>>>> >>>>> >>>>> Quoting RichardWarren >>>> <richard.warren@userite.com<mailto:richard.warren@userite.com>>: >>>>> >>>>> Dear Alistair and All, >>>>> >>>>> Having just spent a fortune getting my son's car through its MOT I >>>>> have to agree with Alistair 100%. Our task is to establish a >>>>> methodology for evaluating website accessibility. If the evaluation >>>>> identifies that the site fully meets the guidelines then a >>>> conformance >>>>> claim can be made to that effect. Everyone will know exactly what >>>> that >>>>> means. >>>>> >>>>> If the site "almost" meets the guidelines then perhaps some other >>>> form >>>>> of "compliance statement" can be made - BUT that is not our current >>>>> problem. Maybe, once we have finished our methodology, we can >>>>> recommend a new task force to look at variance in conformance >>>>> claims<grin>. >>>>> >>>>> >>>>> Regards >>>>> Richard >>>>> >>>>> >>>>> >>>>> -----Original Message----- From: Alistair Garrison >>>>> Sent: Thursday, January 19, 2012 9:02 PM >>>>> To: Eval TF >>>>> Subject: 100% conformance for the pages sampled... >>>>> >>>>> Dear All, >>>>> >>>>> If I understood correctly from this afternoon's EVAL TF telecon - >>>>> there was a suggestion that we should (at a minimum) require the >>>>> representative sample pages to be in 100% conformance with WCAG 2.0 >>>>> (at the chosen level) in order to say the site conforms (at that >>>>> level). If this was the case, I strongly agree with it (meant to >>>>> write it in the IRC at the time). >>>>> >>>>> In addition, I noted from some a worry about telling a website >>>>> owner (a client, etc) that their website doesn't conform - >>>>> especially when they might have tried hard to do so. To my mind, >>>>> worries of this >>>> kind >>>>> should not deter us from asking for nothing less than 100% >>>> conformance >>>>> (on any given sample). The person that does the MOT on my car has >>>>> absolutely no worries about telling me about any failures, but >>>>> possibly that's because everyone doing MOTs requires 100% >>>>> conformance from a car for a pass. >>>>> >>>>> Surely, we want people to try their absolute best to conform 100%. >>>>> We must encourage them to shoot for the stars (100% conformance) - >>>>> some, of course, will initially only hit the moon, but they will at >>>>> least know what is expected from them... Let's not, however, start >>>>> to congratulate people for simply getting off the ground - that >>>>> time >>>> must >>>>> have passed long, long, long ago. >>>>> >>>>> Anyway, look forward to seeing you all on the list. >>>>> >>>>> Alistair >>>>> >>>>> >>>>> >>>>> -- >>>>> --------------------------------------------------------------- >>>>> Detlev Fischer PhD >>>>> DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen >>>>> Geschäftsführung: Thomas Lilienthal, Michael Zapp >>>>> >>>>> Telefon: +49-40-43 18 75-25 >>>>> Mobile: +49-157 7-170 73 84 >>>>> Fax: +49-40-43 18 75-19 >>>>> E-Mail: fischer@dias.de<mailto:fischer@dias.de> >>>>> >>>>> Anschrift: Schulterblatt 36, D-20357 Hamburg Amtsgericht Hamburg >>>>> HRB >>>> 58 167 >>>>> Geschäftsführer: Thomas Lilienthal, Michael Zapp >>>>> --------------------------------------------------------------- >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> = >>> >>> >>> >> >> -- >> Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead, >> W3C/WAI International Program Office Evaluation and Repair Tools >> Working Group (ERT WG) Research and Development Working Group (RDWG) >> >> > > -- > Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead, W3C/WAI International Program Office Evaluation and Repair Tools Working Group (ERT WG) Research and Development Working Group (RDWG) > -- Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead, W3C/WAI International Program Office Evaluation and Repair Tools Working Group (ERT WG) Research and Development Working Group (RDWG)
Received on Tuesday, 31 January 2012 12:32:02 UTC