Re: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]

Hi everyone,

First, Peter, thanks for the draft, which is good, I think. I fully agree that is important to show that WCAG EM is useful and can be used even when a conformance statement for the site under test cannot be made (and may never be successfully made).

As to the wish for more statistical rigor: 

All measures / benchmark points relying on counts (e.g., sampled pages / states vs. total pages / states) will be tricky (possibly dubious) for large and complex sites (sites involving user interactions and generated content). How would an evaluator be able to figure out the total number of permutations of a page based on an initial state and likely user interactions? If, for the sake of quantification of confidence, a tester has to establish such ratios, he/she would have quite a job at hand, and I can already picture the resulting artefacts that you always get when something is 'required' but essentially undoable.

Variability of outcomes across pages is a possible benchmark though I am not sure that it will tell us much - a large offset in terms of degree of compliance could be down to 

* 'forgotten' / legacy content; 
* more complex scripted interactional content,
* user-generated content (failing SC 1.1.1 or 1.3.1)
* the inclusion of external content (social media bookmarks) 
 
and so on. 

I think we must not forget that in most cases the value of WCAG EM results will be in the earmarking of found issues and  consequent recommendations. Statistics will often be dubious (sure, I can see the benefit *even of dubious statistics* especially when repeated checks are made and improvements to the site/content can thus be tracked).

My main worry is that we overload WCAG EM by requirements for statistical analysis that may often be very hard or impossible to meet *if done right*. My guess is that requirements for quantitative  test metadata (measures) that are difficult to meet may be skipped, filled out with artefact data, or worse, frighten off potential users of WCAG EM who think, 'Oh well, this really quite tricky, better leave it alone'.

So my recommendation would be to 'keep it simple' at least all the non-optional stuff, and provide good instructions on how to deal with the non-trivial statistical issues as an optional add-on.

Detlev

On 31 Jul 2013, at 21:58, Peter Korn wrote:

> Frederick,
> 
> Good points.  I do very much hope that we can bring more statistical rigor into this work, so that we could better quantify the likelihood that the larger site that we have sampled is accurately characterized by our report.  That is truly what I mean by "likelihood".
> 
> This likelihood covers the full range of "conformance results".  E.g. if our sample of pages shows 100% support for WCAG 2.0 SCs (e.g. we can make individual "WCAG 2.0 conformance claims" for each and every one of the pages sampled), then the sample size and other measures (variability in the pages) should allow us to calculate the "likelihood" the entire site is capable of making a "WCAG 2.0 conformance claim".
> 
> More more common, I think, is that there will be some number of pages that don't fully support all WCAG 2.0 SCs (and the A/AA/AAA level).  In that case, again the sample size and other measures should allow us to calculate the "likelihood" that the entire site has "a similar level of support for WCAG 2.0 SCs".  E.g. that the entire site is "this bad" (or conversely, "this good, but not perfect").
> 
> 
> Peter
> 
> On 7/31/2013 12:33 PM, Boland Jr, Frederick E. wrote:
>> Thanks for doing this!
>>  
>> In WCAG2.0, as you mention, “conformance is defined only for web pages“ (according to their definition of web page):
>> http://www.w3.org/TR/WCAG/#conformance
>>  
>> , so I’m not sure, as a result of applying WCAG-EM, that we can talk objectively about “extent” or “likelihood” of WCAG2.0 conformance for “websites” or “web applications”, if they are different from WCAG2.0 definition of “web page” .    We can discuss WCAG2.0 conformance for each of the specific web pages sampled, but I think going beyond this in an objective way may be considered conjecture or speculation,  unless there is some mathematical or statistical basis (asserting confidence level from a statistical distribution or repeated sampling, for example)  for making that kind of assertion..    if there was a way to quantify “extent” or “likelihood”, that would be progress..
>>  
>> From: Peter Korn [mailto:peter.korn@oracle.com] 
>> Sent: Wednesday, July 31, 2013 2:35 PM
>> To: Velleman, Eric
>> Cc: public-wai-evaltf@w3.org
>> Subject: Proposed text on "WCAG-EM Conformance" [was Re: EvalTF Telco]
>>  
>> Hi Eric, all,
>> 
>> At our last meeting we discussed the fact that WCAG 2.0 defines a conformance claim "only for Web pages", and that it can only be made if a page fully satisfies all of the success criteria (A/AA/AAA as appropriate to the claim).  We also observed that since WCAG-EM defines a sampling methodology, the output of a WCAG-EM evaluation couldn't be a "WCAG 2.0 conformance claim" for the website being evaluated, since by the nature of sampling not all pages had been evaluated.  And finally, we agreed that a new section describing this situation and what conformance means for WCAG-EM should be added.
>> 
>> I took the action from that meeting to draft proposed language for that section, which I include below.  Perhaps if there is time we might start discussing this tomorrow at our meeting.  Looking at our current editors draft, I suggest this new section should go into the Introduction section, appearing right after Terms and Definitions.
>> 
>> ================================
>> "Conformance" in the context of WCAG-EM
>> 
>> WCAG 2.0 defines the term "conformance", and also describes the requirements for a Web page to conform to WCAG 2.0.  Furthermore, WCAG 2.0 describes the requirements of an optional WCAG 2.0 conformance claim, which "is defined only for Web pages".  Under these terms and descriptions, a claim may only be made for specific pages which are explicitly known to satisfy all of the requirements (at the A, AA, or AAA level as appropriate).  This means a "WCAG 2.0 conformance claim" cannot be made for a collection of web pages unless it is known that all of those web pages satisfy all of the requirements.
>> 
>> The purpose of WCAG-EM is to define a methodology for sampling and evaluating large or complex websites and web applications using the WCAG 2.0 success criteria (at the A, AA, and AAA levels as appropriate), based on a sample of that website / web application, without evaluating all of the web pages or possible user interactions with a web application.  This is often necessary because it is rarely if ever possible to exhaustively evaluate every page for every success criteria in large or complex websites, or to evaluate every possible user input and dynamic output of large or complex web applications.
>> 
>> As a result of this, WCAG-EM cannot be used to make a "WCAG 2.0 conformance claim" for a website or web application, as only a subset of that website or web application has been evaluated.  When used successfully, however, WCAG-EM can ascertain the extent to which a website or web application likely conforms.  This result of WCAG-EM can then be used to generate a "WCAG 2.0 conformance report" (see Appendix C: Example Reports for the kinds of reports that can be generated).  The more detailed WCAG-EM reports described herein note the issues found in the evaluation, and provide some confidence level as to how representative these findings are for the entire website or web application under evaluation.
>> 
>> ================================
>> 
>> 
>> Regards,
>> 
>> Peter
>> 
>> On 7/23/2013 2:14 PM, Velleman, Eric wrote:
>> Dear Eval TF,
>>  
>> The next teleconference is scheduled for Thursday 25 July 2013 at:
>>   * 14:00 to 15:00 UTC
>>   * 15:00 to 16:00 UK Time
>>   * 16:00 to 17:00 Central European Time (time we use as reference)
>>   * 10:00 to 11:00 North American Eastern Time (ET)
>>   * 07:00 to 08:00 North American Pacific Time (PT)
>>   * 22:00 to 23:00 Western Australia Time
>>   * 00:00 to 01:00 Eastern Australia Time (Note: Friday)
>>  
>> Please check the World Clock Meeting Planner to find out the precise date for your own time zone:
>>   - <http://www.timeanddate.com/worldclock/meeting.html>
>>  
>> The teleconference information is: (Passcode 3825 - "EVAL")
>>   * +1.617.761.6200
>>   * SIP / VoIP -http://www.w3.org/2006/tools/wiki/Zakim-SIP
>>  
>> We also use IRC to support the meeting: (http://irc.w3.org)
>>   * IRC server: irc.w3.org
>>   * port: 6665
>>   * channel: #eval
>>  
>>  
>> AGENDA:
>>  
>> #1. Welcome
>>  
>> #2. Status of Survey 10
>> Survey: https://www.w3.org/2002/09/wbs/48225/evaltfq10/
>> What is accepted and what do we need to discuss further. Topics will be sent just before the telco and discussed in next agenda item.
>>  
>> #3. Discuss topics from the survey and decide if they need to be further discussed on the list
>>  
>> #4. Parameters of the sample size
>> See discussion at: http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0028.html
>>  
>> #5. Other issues
>>  
>> Kindest regards,
>> Eric
>>  
>>  
>> -- 
>> <Mail Attachment.gif>
>> Peter Korn | Accessibility Principal
>> Phone: +1 650 5069522 
>> 500 Oracle Parkway | Redwood City, CA 94064 
>> <Mail Attachment.gif>Oracle is committed to developing practices and products that help protect the environment
>> 
> 
> -- 
> <oracle_sig_logo.gif>
> Peter Korn | Accessibility Principal
> Phone: +1 650 5069522 
> 500 Oracle Parkway | Redwood City, CA 94064 
> <green-for-email-sig_0.gif> Oracle is committed to developing practices and products that help protect the environment

-- 
Detlev Fischer
testkreis - das Accessibility-Team von feld.wald.wiese
c/o feld.wald.wiese
Thedestraße 2
22767 Hamburg

Tel   +49 (0)40 439 10 68-3
Mobil +49 (0)1577 170 73 84
Fax   +49 (0)40 439 10 68-5

http://www.testkreis.de
Beratung, Tests und Schulungen für barrierefreie Websites

Received on Thursday, 1 August 2013 06:52:21 UTC