RE: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]

Hi all
I fully agree with what both Federick and Peter are saying.  It is important that we consider the statistical rigour of our work.  This is where the correctness of our sampling plays a huge part.  I'm not sure how we'd go about doing the calculations though that Peter mentions and would love to hear more of how you think this would work.


Regards

Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(CS)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au<mailto:v.conway@ecu.edu.au>
v.conway@webkeyit.com<mailto:v.conway@webkeyit.com>
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.

________________________________
From: Peter Korn [peter.korn@oracle.com]
Sent: Thursday, 1 August 2013 3:58 AM
To: Boland Jr, Frederick E.
Cc: Velleman, Eric; public-wai-evaltf@w3.org
Subject: Re: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]

Frederick,

Good points.  I do very much hope that we can bring more statistical rigor into this work, so that we could better quantify the likelihood that the larger site that we have sampled is accurately characterized by our report.  That is truly what I mean by "likelihood".

This likelihood covers the full range of "conformance results".  E.g. if our sample of pages shows 100% support for WCAG 2.0 SCs (e.g. we can make individual "WCAG 2.0 conformance claims" for each and every one of the pages sampled), then the sample size and other measures (variability in the pages) should allow us to calculate the "likelihood" the entire site is capable of making a "WCAG 2.0 conformance claim".

More more common, I think, is that there will be some number of pages that don't fully support all WCAG 2.0 SCs (and the A/AA/AAA level).  In that case, again the sample size and other measures should allow us to calculate the "likelihood" that the entire site has "a similar level of support for WCAG 2.0 SCs".  E.g. that the entire site is "this bad" (or conversely, "this good, but not perfect").


Peter

On 7/31/2013 12:33 PM, Boland Jr, Frederick E. wrote:
Thanks for doing this!

In WCAG2.0, as you mention, “conformance is defined only for web pages“ (according to their definition of web page):
http://www.w3.org/TR/WCAG/#conformance

, so I’m not sure, as a result of applying WCAG-EM, that we can talk objectively about “extent” or “likelihood” of WCAG2.0 conformance for “websites” or “web applications”, if they are different from WCAG2.0 definition of “web page” .    We can discuss WCAG2.0 conformance for each of the specific web pages sampled, but I think going beyond this in an objective way may be considered conjecture or speculation,  unless there is some mathematical or statistical basis (asserting confidence level from a statistical distribution or repeated sampling, for example)  for making that kind of assertion..    if there was a way to quantify “extent” or “likelihood”, that would be progress..

From: Peter Korn [mailto:peter.korn@oracle.com]
Sent: Wednesday, July 31, 2013 2:35 PM
To: Velleman, Eric
Cc: public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org>
Subject: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]

Hi Eric, all,

At our last meeting we discussed the fact that WCAG 2.0 defines a conformance claim "only for Web pages", and that it can only be made if a page fully satisfies all of the success criteria (A/AA/AAA as appropriate to the claim).  We also observed that since WCAG-EM defines a sampling methodology, the output of a WCAG-EM evaluation couldn't be a "WCAG 2.0 conformance claim" for the website being evaluated, since by the nature of sampling not all pages had been evaluated.  And finally, we agreed that a new section describing this situation and what conformance means for WCAG-EM should be added.

I took the action from that meeting to draft proposed language for that section, which I include below.  Perhaps if there is time we might start discussing this tomorrow at our meeting.  Looking at our current editors draft<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712>, I suggest this new section should go into the Introduction section<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#introduction>, appearing right after Terms and Definitions<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#terms>.

================================
"Conformance" in the context of WCAG-EM
WCAG 2.0<http://www.w3.org/TR/WCAG> defines the term "conformance<http://www.w3.org/TR/WCAG/#conformancedef>", and also describes the requirements for a Web page to conform to WCAG 2.0<http://www.w3.org/TR/WCAG/#conformance-reqs>.  Furthermore, WCAG 2.0 describes the requirements of an optional WCAG 2.0 conformance claim<http://www.w3.org/TR/WCAG/#conformance-claims>, which "is defined only for Web pages".  Under these terms and descriptions, a claim may only be made for specific pages which are explicitly known to satisfy all of the requirements (at the A, AA, or AAA level as appropriate).  This means a "WCAG 2.0 conformance claim" cannot be made for a collection of web pages unless it is known that all of those web pages satisfy all of the requirements.

The purpose of WCAG-EM is to define a methodology for sampling and evaluating large or complex websites and web applications using the WCAG 2.0 success criteria (at the A, AA, and AAA levels as appropriate), based on a sample of that website / web application, without evaluating all of the web pages or possible user interactions with a web application.  This is often necessary because it is rarely if ever possible to exhaustively evaluate every page for every success criteria in large or complex websites, or to evaluate every possible user input and dynamic output of large or complex web applications.

As a result of this, WCAG-EM cannot be used to make a "WCAG 2.0 conformance claim" for a website or web application, as only a subset of that website or web application has been evaluated.  When used successfully, however, WCAG-EM can ascertain the extent to which a website or web application likely conforms.  This result of WCAG-EM can then be used to generate a "WCAG 2.0 conformance report" (see Appendix C: Example Reports<http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#reports> for the kinds of reports that can be generated).  The more detailed WCAG-EM reports described herein note the issues found in the evaluation, and provide some confidence level as to how representative these findings are for the entire website or web application under evaluation.

================================


Regards,

Peter
On 7/23/2013 2:14 PM, Velleman, Eric wrote:

Dear Eval TF,



The next teleconference is scheduled for Thursday 25 July 2013 at:

  * 14:00 to 15:00 UTC

  * 15:00 to 16:00 UK Time

  * 16:00 to 17:00 Central European Time (time we use as reference)

  * 10:00 to 11:00 North American Eastern Time (ET)

  * 07:00 to 08:00 North American Pacific Time (PT)

  * 22:00 to 23:00 Western Australia Time

  * 00:00 to 01:00 Eastern Australia Time (Note: Friday)



Please check the World Clock Meeting Planner to find out the precise date for your own time zone:

  - <http://www.timeanddate.com/worldclock/meeting.html><http://www.timeanddate.com/worldclock/meeting.html>



The teleconference information is: (Passcode 3825 - "EVAL")

  * +1.617.761.6200

  * SIP / VoIP -http://www.w3.org/2006/tools/wiki/Zakim-SIP



We also use IRC to support the meeting: (http://irc.w3.org)

  * IRC server: irc.w3.org

  * port: 6665

  * channel: #eval





AGENDA:



#1. Welcome



#2. Status of Survey 10

Survey: https://www.w3.org/2002/09/wbs/48225/evaltfq10/

What is accepted and what do we need to discuss further. Topics will be sent just before the telco and discussed in next agenda item.



#3. Discuss topics from the survey and decide if they need to be further discussed on the list



#4. Parameters of the sample size

See discussion at: http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0028.html



#5. Other issues



Kindest regards,

Eric



--
[cid:part14.01000308.00090006@oracle.com]<http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522<tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94064
[cid:part17.04080908.00020601@oracle.com]<http://www.oracle.com/commitment>Oracle is committed to developing practices and products that help protect the environment

--
[cid:part19.01000400.01060805@oracle.com]<http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522<tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94064
[cid:part22.09030702.00000600@oracle.com]<http://www.oracle.com/commitment> Oracle is committed to developing practices and products that help protect the environment

________________________________
This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B