AW: 100% conformance for the pages sampled...

Hi Eric, all,

> Thanks. I would like to avoid the word violations, but we could change
> the title to something like margin for failures or failure tolerance.
> Please feel free to propose a better term.

What about "expected degree of conformance". In the text we can say: no
degrees, 100% for the sample.
 
> We have agreed that the methodology will have a 0% margin for failures.
> This seems more strict than is normal in scientific research where
> mostly an error margin of 5 percent is accepted for the sample.

When we just have a look on failures. As written in another mail we should
be careful with terms. Error margin is very close to "Margin of error".
Wikipedia gives a short definition: "The margin of error is a statistic
expressing the amount of random sampling error in a survey's results. The
larger the margin of error, the less faith one should have that the poll's
reported results are close to the "true" figures; that is, the figures for
the whole population. Margin of error occurs whenever a population is
incompletely sampled." Or

Margin of error occurs whenever a website is incompletely sampled.
 
> Agree that we also need to describe the possibility of sampling errors
> like you propose (that would then be errors made in selecting the
> sample, by tools or manual). Shall we add it to the sampling section?

I think, for the moment and we decide later upon this.

I'm no master of statistics and it's a couple of years ago that I was
bothered with test theory, scaling and all this. But I'm a bit concerned
about our terminology. Shouldn't we collect resources with understandable
explanations/definitions of specific terms in a Wiki? This might also be
important for the glossary of our methodology, especially if we are using
specific terms in another way (which we should avoid in general) and also
because of future translations.

Best

Kerstin

 
> Kindest regards,
> 
> Eric
> 
> 
> ________________________________________
> Van: Kerstin Probiesch [k.probiesch@googlemail.com]
> Verzonden: vrijdag 27 januari 2012 7:06
> Aan: Velleman, Eric
> CC: Alistair Garrison; EvalTF
> Onderwerp: Re: 100% conformance for the pages sampled...
> 
> Hi Eric, all
> 
> I still don't agree with the section "Error margin". As pointed out:
> error margin a specific term. We should not use this term, when we
> speak about failures on pages.
> 
> To make clear what we mean, we should use something like "failures" or
> "violations..." as heading for this section. Nevertheless we also need
> a section "error margin", where we describe possible systematic errors
> like sampling errors. I remember that we discussed also the term
> "systematic error" just in the context of failures on pages. As this an
> evaluation methodology I believe we should be careful with terms and
> shouldn't use specific terms of testing theories and methodologies in
> general for anything else to avoid confusion.
> 
> Best
> 
> Kerstin
> 
> Am 27.01.2012 um 01:18 schrieb "Velleman, Eric"
> <evelleman@bartimeus.nl>:
> 
> > Hi Alistair,
> >
> > The 100% conformance you refer to in your mail made it into the
> document in clause 5.5 Error Margin as we decided it: "The Methodology
> assumes a strict conformance requirement for all resources in the
> sample. I could change strict into 100%."
> >
> > Also in the same section I added a disclaimer following the outcome
> of our discussion. We said that it is necessary to say that we cannot
> guarantee that there will not be another error somewhere on the site.
> This is caused by the use of a sample: "It is important to note that
> due to the use of a sample there is never a 100% guarantee that the
> full website is conformant."
> >
> > I thought it might be good to explain that further and I propose to
> do that in the Random Resources clause (4.1.3). I already made a short
> link: "This specific error margin for a given confidence level is
> described in the section about Random Resources." We will discuss
> clause 4.1 next week (as agreed today), so we can then discuss if we
> want to keep that in the document or take it out again.
> >
> > Kindest regards,
> >
> > Eric
> >
> >
> > ________________________________
> > Van: Alistair Garrison [alistair.j.garrison@gmail.com]
> > Verzonden: donderdag 26 januari 2012 15:50
> > Aan: Eval TF
> > Onderwerp: Fwd: 100% conformance for the pages sampled...
> >
> > Dear All,
> >
> > The concept of "100% conformance for the pages sampled" gained a
> reasonable level of consensus amongst EVAL TF members (6 or 7 people).
> Several other ideas have seemingly gained the same group approval, but
> then fallen by the wayside as we've move ahead to debate something
> else...
> >
> > I'd like to know, what mechanism are we following to capture such
> ideas, and ensure that they are appear in 'black & white' in our
> methodology updates?
> >
> > All the best
> >
> > Alistair
> >
> > Begin forwarded message:
> >
> > From: Alistair Garrison
> <alistair.j.garrison@gmail.com<mailto:alistair.j.garrison@gmail.com>>
> > Date: 20 January 2012 14:10:20 CET
> > To: Léonie Watson <lwatson@nomensa.com<mailto:lwatson@nomensa.com>>,
> Eval TF <public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org>>
> > Subject: Re: 100% conformance for the pages sampled...
> >
> > Hi Léonie,
> >
> >> From http://www.w3.org/TR/WCAG/#conformance-claims - "Conformance is
> not possible at a particular level if any page in the process does not
> conform at that level or better".
> >
> > And, as an additional component of the conformance claim "A list of
> success criteria beyond the level of conformance claimed that have been
> met. This information should be provided in a form that users can use,
> preferably machine-readable metadata."
> >
> > So, I would suggest that for our purposes (those of evaluating the
> whole site) this would mean - making a level AA conformance claim for
> all pages in the sample (website).  If required by the website owner
> (or other) it could be provided along with a list of urls for the web
> pages on which the additional AAA Success Criteria have been achieved.
> >
> > If we were not looking at the whole site separate AA and AAA claims
> for the relevant pages would probably be the way to go.  I should add,
> that a AAA conformance claim could only be made for the pages which
> were AAA conformant - ruling out, in my opinion, a AAA with exceptions
> type claim.
> >
> > Let me know what you think.
> >
> > All the best
> >
> > Alistair
> >
> > On 20 Jan 2012, at 12:59, Léonie Watson wrote:
> >
> >    Thanks Alastair, it does. Although it's started me thinking more
> about this. Given that WCAG acknowledges that Level AAA is not possible
> with all technologies, we could end up with a sample that is partly
> Level AA compliant and partly Level AAA compliant.
> >
> >    I'm thinking out loud here as much as anything, but would we
> therefore consider the sample to be 100% Level AA compliant with
> additional achievements, Level AAA compliant with exceptions, or part
> Level AA and part Level AAA as applicable?
> >
> >
> > Léonie.
> >
> > --
> > Nomensa - humanising technology
> >
> > Léonie Watson, Director of Accessibility & Web Development
> > tel: +44 (0)117 929 7333
> > mob: +44 (0)792 116 8551
> > twitter: @we_are_Nomensa @LeonieWatson
> >
> > Nomensa Email Disclaimer: http://www.nomensa.com/email-disclaimer
> >
> >
> >
> > ________________________________
> > From: Alistair Garrison [mailto:alistair.j.garrison@gmail.com]
> > Sent: 20 January 2012 09:38
> > To: Eval TF
> > Subject: Re: 100% conformance for the pages sampled...
> >
> > Hi Leonie,
> >
> >> From http://www.w3.org/TR/WCAG/#conformance-claims - "Conformance is
> defined only for Web pages. However, a conformance claim may be made to
> cover one page, a series of pages, or multiple related Web pages."
> >
> > Where they say "for Web Pages" I think they meant to say "for full
> Web Pages" - referencing "Conformance (and conformance level) is for
> full Web page(s) only, and cannot be achieved if part of a Web page is
> excluded." from the same page.
> >
> > Hope this clarifies things...
> >
> > All the best
> >
> > Alistair
> >
> > On 20 Jan 2012, at 10:14, Léonie Watson wrote:
> >
> > If I understand correctly, WCAG conformance relates to a single page,
> not to an entire website. I'd be interested in people's thoughts on how
> this might (or might not) influence our thinking about an error margin?
> >
> >
> > Léonie.
> >
> > --
> > Nomensa - humanising technology
> >
> > Léonie Watson, Director of Accessibility & Web Development
> > tel: +44 (0)117 929 7333
> > mob: +44 (0)792 116 8551
> > twitter: @we_are_Nomensa @LeonieWatson
> >
> > Nomensa Email Disclaimer: http://www.nomensa.com/email-disclaimer
> >
> > -----Original Message-----
> > From: Detlev Fischer [mailto:fischer@dias.de]
> > Sent: 19 January 2012 21:58
> > To: public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org>
> > Subject: Re: 100% conformance for the pages sampled...
> >
> > Let's stop here and consider the implications.
> >
> > Here and then, people in the EVAL TF have agreed that the 100%
> conformant site does not really exist 'out there'. Aren't we holding
> the bone a wee bit too high? I wonder what that will mean for the
> practical acceptance of the methodology. Will it come to be derided as
> academic, as impossibly demanding? Who then is the customer of a
> (sorry, chap) refused seal of conformance who bows to gracefully accept
> the list of flaws to rectify? Just wondering...it just strikes me as
> slightly surreal...
> >
> > Detlev
> >
> >
> >
> > Quoting RichardWarren
> <richard.warren@userite.com<mailto:richard.warren@userite.com>>:
> >
> > Dear Alistair and All,
> >
> > Having just spent a fortune getting my son's car through its MOT I
> > have to agree with Alistair 100%. Our task is to establish a
> > methodology for evaluating website accessibility. If the evaluation
> > identifies that the site fully meets the guidelines then a
> conformance
> > claim can be made to that effect. Everyone will know exactly what
> that
> > means.
> >
> > If the site "almost" meets the guidelines then perhaps some other
> form
> > of "compliance statement" can be made - BUT that is not our current
> > problem. Maybe, once we have finished our methodology, we can
> > recommend a new task force to look at variance in conformance claims
> > <grin>.
> >
> >
> > Regards
> > Richard
> >
> >
> >
> > -----Original Message----- From: Alistair Garrison
> > Sent: Thursday, January 19, 2012 9:02 PM
> > To: Eval TF
> > Subject: 100% conformance for the pages sampled...
> >
> > Dear All,
> >
> > If I understood correctly from this afternoon's EVAL TF telecon -
> > there was a suggestion that we should (at a minimum) require the
> > representative sample pages to be in 100% conformance with WCAG 2.0
> > (at the chosen level) in order to say the site conforms (at that
> > level).  If this was the case, I strongly agree with it (meant to
> > write it in the IRC at the time).
> >
> > In addition, I noted from some a worry about telling a website owner
> > (a client, etc) that their website doesn't conform - especially when
> > they might have tried hard to do so.  To my mind, worries of this
> kind
> > should not deter us from asking for nothing less than 100%
> conformance
> > (on any given sample). The person that does the MOT on my car has
> > absolutely no worries about telling me about any failures, but
> > possibly that's because everyone doing MOTs requires 100% conformance
> > from a car for a pass.
> >
> > Surely, we want people to try their absolute best to conform 100%.
> > We must encourage them to shoot for the stars (100% conformance) -
> > some, of course, will initially only hit the moon, but they will at
> > least know what is expected from them... Let's not, however, start to
> > congratulate people for simply getting off the ground - that time
> must
> > have passed long, long, long ago.
> >
> > Anyway, look forward to seeing you all on the list.
> >
> > Alistair
> >
> >
> >
> > --
> > ---------------------------------------------------------------
> > Detlev Fischer PhD
> > DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
> > Geschäftsführung: Thomas Lilienthal, Michael Zapp
> >
> > Telefon: +49-40-43 18 75-25
> > Mobile: +49-157 7-170 73 84
> > Fax: +49-40-43 18 75-19
> > E-Mail: fischer@dias.de<mailto:fischer@dias.de>
> >
> > Anschrift: Schulterblatt 36, D-20357 Hamburg Amtsgericht Hamburg HRB
> 58 167
> > Geschäftsführer: Thomas Lilienthal, Michael Zapp
> > ---------------------------------------------------------------
> >
> >
> >
> >
> >
> >
> 
> =

Received on Sunday, 29 January 2012 10:11:31 UTC