- From: Peter Korn <peter.korn@oracle.com>
- Date: Wed, 31 Jul 2013 12:58:13 -0700
- To: "Boland Jr, Frederick E." <frederick.boland@nist.gov>
- CC: "Velleman, Eric" <evelleman@bartimeus.nl>, "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>
- Message-ID: <51F96C55.1090907@oracle.com>
Frederick, Good points. I do very much hope that we can bring more statistical rigor into this work, so that we could better quantify the likelihood that the larger site that we have sampled is accurately characterized by our report. That is truly what I mean by "likelihood". This likelihood covers the full range of "conformance results". E.g. if our sample of pages shows 100% support for WCAG 2.0 SCs (e.g. we can make individual "WCAG 2.0 conformance claims" for each and every one of the pages sampled), then the sample size and other measures (variability in the pages) should allow us to calculate the "likelihood" the entire site is capable of making a "WCAG 2.0 conformance claim". More more common, I think, is that there will be some number of pages that don't fully support all WCAG 2.0 SCs (and the A/AA/AAA level). In that case, again the sample size and other measures should allow us to calculate the "likelihood" that the entire site has "a similar level of support for WCAG 2.0 SCs". E.g. that the entire site is "this bad" (or conversely, "this good, but not perfect"). Peter On 7/31/2013 12:33 PM, Boland Jr, Frederick E. wrote: > > Thanks for doing this! > > In WCAG2.0, as you mention, "conformance is defined only for web > pages" (according to their definition of web page): > > http://www.w3.org/TR/WCAG/#conformance > > , so I'm not sure, as a result of applying WCAG-EM, that we can talk > objectively about "extent" or "likelihood" of WCAG2.0 conformance for > "websites" or "web applications", if they are different from WCAG2.0 > definition of "web page" . We can discuss WCAG2.0 conformance for > each of the specific web pages sampled, but I think going beyond this > in an objective way may be considered conjecture or speculation, > unless there is some mathematical or statistical basis (asserting > confidence level from a statistical distribution or repeated sampling, > for example) for making that kind of assertion.. if there was a > way to quantify "extent" or "likelihood", that would be progress.. > > *From:*Peter Korn [mailto:peter.korn@oracle.com] > *Sent:* Wednesday, July 31, 2013 2:35 PM > *To:* Velleman, Eric > *Cc:* public-wai-evaltf@w3.org > *Subject:* Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco] > > Hi Eric, all, > > At our last meeting we discussed the fact that WCAG 2.0 defines a > conformance claim "only for Web pages", and that it can only be made > if a page fully satisfies /all/ of the success criteria (A/AA/AAA as > appropriate to the claim). We also observed that since WCAG-EM > defines a sampling methodology, the output of a WCAG-EM evaluation > couldn't be a "WCAG 2.0 conformance claim" for the website being > evaluated, since by the nature of sampling not all pages had been > evaluated. And finally, we agreed that a new section describing this > situation and what conformance means for WCAG-EM should be added. > > I took the action from that meeting to draft proposed language for > that section, which I include below. Perhaps if there is time we > might start discussing this tomorrow at our meeting. Looking at our > current editors draft > <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712>, I > suggest this new section should go into the Introduction section > <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#introduction>, > appearing right after Terms and Definitions > <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#terms>. > > ================================ > > > "Conformance" in the context of WCAG-EM > > WCAG 2.0 <http://www.w3.org/TR/WCAG> defines the term "conformance > <http://www.w3.org/TR/WCAG/#conformancedef>", and also describes the > requirements for a Web page to conform to WCAG 2.0 > <http://www.w3.org/TR/WCAG/#conformance-reqs>. Furthermore, WCAG 2.0 > describes the requirements of an optional WCAG 2.0 conformance claim > <http://www.w3.org/TR/WCAG/#conformance-claims>, which "is defined > only for Web pages". Under these terms and descriptions, a claim may > only be made for specific pages which are explicitly known to satisfy > /all of the requirements/ (at the A, AA, or AAA level as > appropriate). This means a "WCAG 2.0 conformance claim" cannot be > made for a collection of web pages unless it is known that /all/ of > those web pages satisfy /all /of the requirements. > > The purpose of WCAG-EM is to define a methodology for sampling and > evaluating large or complex websites and web applications using the > WCAG 2.0 success criteria (at the A, AA, and AAA levels as > appropriate), based on a sample of that website / web application, > /without /evaluating /all /of the web pages or possible user > interactions with a web application. This is often necessary because > it is rarely if ever possible to exhaustively evaluate every page for > every success criteria in large or complex websites, or to evaluate > every possible user input and dynamic output of large or complex web > applications. > > As a result of this, WCAG-EM cannot be used to make a "WCAG 2.0 > conformance claim" for a website or web application, as only a subset > of that website or web application has been evaluated. When used > successfully, however, WCAG-EM can ascertain the /extent to which a > website or web application *likely *conforms/. This result of WCAG-EM > can then be used to generate a "WCAG 2.0 conformance report" (see > Appendix C: Example Reports > <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#reports> > for the kinds of reports that can be generated). The more detailed > WCAG-EM reports described herein note the issues found in the > evaluation, and provide some confidence level as to how representative > these findings are for the entire website or web application under > evaluation. > > ================================ > > > Regards, > > Peter > > On 7/23/2013 2:14 PM, Velleman, Eric wrote: > > Dear Eval TF, > > > > The next teleconference is scheduled for Thursday 25 July 2013 at: > > * 14:00 to 15:00 UTC > > * 15:00 to 16:00 UK Time > > * 16:00 to 17:00 Central European Time (time we use as reference) > > * 10:00 to 11:00 North American Eastern Time (ET) > > * 07:00 to 08:00 North American Pacific Time (PT) > > * 22:00 to 23:00 Western Australia Time > > * 00:00 to 01:00 Eastern Australia Time (Note: Friday) > > > > Please check the World Clock Meeting Planner to find out the precise date for your own time zone: > > -<http://www.timeanddate.com/worldclock/meeting.html> <http://www.timeanddate.com/worldclock/meeting.html> > > > > The teleconference information is: (Passcode 3825 - "EVAL") > > * +1.617.761.6200 > > * SIP / VoIP -http://www.w3.org/2006/tools/wiki/Zakim-SIP > > > > We also use IRC to support the meeting: (http://irc.w3.org) > > * IRC server: irc.w3.org > > * port: 6665 > > * channel: #eval > > > > > > AGENDA: > > > > #1. Welcome > > > > #2. Status of Survey 10 > > Survey:https://www.w3.org/2002/09/wbs/48225/evaltfq10/ > > What is accepted and what do we need to discuss further. Topics will be sent just before the telco and discussed in next agenda item. > > > > #3. Discuss topics from the survey and decide if they need to be further discussed on the list > > > > #4. Parameters of the sample size > > See discussion at:http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0028.html > > > > #5. Other issues > > > > Kindest regards, > > Eric > > > > -- > Oracle <http://www.oracle.com> > Peter Korn | Accessibility Principal > Phone: +1 650 5069522 <tel:+1%20650%205069522> > 500 Oracle Parkway | Redwood City, CA 94064 > Green Oracle <http://www.oracle.com/commitment>Oracle is committed to > developing practices and products that help protect the environment > -- Oracle <http://www.oracle.com> Peter Korn | Accessibility Principal Phone: +1 650 5069522 <tel:+1%20650%205069522> 500 Oracle Parkway | Redwood City, CA 94064 Green Oracle <http://www.oracle.com/commitment> Oracle is committed to developing practices and products that help protect the environment
Received on Wednesday, 31 July 2013 19:59:30 UTC