RE: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]

Thanks for doing this!

In WCAG2.0, as you mention, "conformance is defined only for web pages" (according to their definition of web page):
http://www.w3.org/TR/WCAG/#conformance

, so I'm not sure, as a result of applying WCAG-EM, that we can talk objectively about "extent" or "likelihood" of WCAG2.0 conformance for "websites" or "web applications", if they are different from WCAG2.0 definition of "web page" .    We can discuss WCAG2.0 conformance for each of the specific web pages sampled, but I think going beyond this in an objective way may be considered conjecture or speculation,  unless there is some mathematical or statistical basis (asserting confidence level from a statistical distribution or repeated sampling, for example)  for making that kind of assertion..    if there was a way to quantify "extent" or "likelihood", that would be progress..

From: Peter Korn [mailto:peter.korn@oracle.com]
Sent: Wednesday, July 31, 2013 2:35 PM
To: Velleman, Eric
Cc: public-wai-evaltf@w3.org
Subject: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]

Hi Eric, all,

At our last meeting we discussed the fact that WCAG 2.0 defines a conformance claim "only for Web pages", and that it can only be made if a page fully satisfies all of the success criteria (A/AA/AAA as appropriate to the claim).  We also observed that since WCAG-EM defines a sampling methodology, the output of a WCAG-EM evaluation couldn't be a "WCAG 2.0 conformance claim" for the website being evaluated, since by the nature of sampling not all pages had been evaluated.  And finally, we agreed that a new section describing this situation and what conformance means for WCAG-EM should be added.

I took the action from that meeting to draft proposed language for that section, which I include below.  Perhaps if there is time we might start discussing this tomorrow at our meeting.  Looking at our current editors draft<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712>, I suggest this new section should go into the Introduction section<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#introduction>, appearing right after Terms and Definitions<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#terms>.

================================
"Conformance" in the context of WCAG-EM
WCAG 2.0<http://www.w3.org/TR/WCAG> defines the term "conformance<http://www.w3.org/TR/WCAG/#conformancedef>", and also describes the requirements for a Web page to conform to WCAG 2.0<http://www.w3.org/TR/WCAG/#conformance-reqs>.  Furthermore, WCAG 2.0 describes the requirements of an optional WCAG 2.0 conformance claim<http://www.w3.org/TR/WCAG/#conformance-claims>, which "is defined only for Web pages".  Under these terms and descriptions, a claim may only be made for specific pages which are explicitly known to satisfy all of the requirements (at the A, AA, or AAA level as appropriate).  This means a "WCAG 2.0 conformance claim" cannot be made for a collection of web pages unless it is known that all of those web pages satisfy all of the requirements.

The purpose of WCAG-EM is to define a methodology for sampling and evaluating large or complex websites and web applications using the WCAG 2.0 success criteria (at the A, AA, and AAA levels as appropriate), based on a sample of that website / web application, without evaluating all of the web pages or possible user interactions with a web application.  This is often necessary because it is rarely if ever possible to exhaustively evaluate every page for every success criteria in large or complex websites, or to evaluate every possible user input and dynamic output of large or complex web applications.

As a result of this, WCAG-EM cannot be used to make a "WCAG 2.0 conformance claim" for a website or web application, as only a subset of that website or web application has been evaluated.  When used successfully, however, WCAG-EM can ascertain the extent to which a website or web application likely conforms.  This result of WCAG-EM can then be used to generate a "WCAG 2.0 conformance report" (see Appendix C: Example Reports<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#reports> for the kinds of reports that can be generated).  The more detailed WCAG-EM reports described herein note the issues found in the evaluation, and provide some confidence level as to how representative these findings are for the entire website or web application under evaluation.

================================


Regards,

Peter
On 7/23/2013 2:14 PM, Velleman, Eric wrote:

Dear Eval TF,



The next teleconference is scheduled for Thursday 25 July 2013 at:

  * 14:00 to 15:00 UTC

  * 15:00 to 16:00 UK Time

  * 16:00 to 17:00 Central European Time (time we use as reference)

  * 10:00 to 11:00 North American Eastern Time (ET)

  * 07:00 to 08:00 North American Pacific Time (PT)

  * 22:00 to 23:00 Western Australia Time

  * 00:00 to 01:00 Eastern Australia Time (Note: Friday)



Please check the World Clock Meeting Planner to find out the precise date for your own time zone:

  - <http://www.timeanddate.com/worldclock/meeting.html><http://www.timeanddate.com/worldclock/meeting.html>



The teleconference information is: (Passcode 3825 - "EVAL")

  * +1.617.761.6200

  * SIP / VoIP -http://www.w3.org/2006/tools/wiki/Zakim-SIP



We also use IRC to support the meeting: (http://irc.w3.org)

  * IRC server: irc.w3.org

  * port: 6665

  * channel: #eval





AGENDA:



#1. Welcome



#2. Status of Survey 10

Survey: https://www.w3.org/2002/09/wbs/48225/evaltfq10/

What is accepted and what do we need to discuss further. Topics will be sent just before the telco and discussed in next agenda item.



#3. Discuss topics from the survey and decide if they need to be further discussed on the list



#4. Parameters of the sample size

See discussion at: http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0028.html



#5. Other issues



Kindest regards,

Eric



--
[Oracle]<http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522<tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94064
[Green            Oracle]<http://www.oracle.com/commitment>Oracle is committed to developing practices and products that help protect the environment