Re: acceptance criteria for new success criteria

Great start

Some comments/edits to the suggested list



gregg

> On Jun 1, 2016, at 5:36 AM, josh@interaccess.ie wrote:
> 
> Hi all,
>  
> Rather than start a new thread, I will add to this. On yesterdays call, we discussed what should be the requirements for creating new SCs.
>  
> The current list (with many thanks to Lisa/David Mc D for their input) is:
>  
> Ensure that requirements may be applied across technologies such as HTML, CSS, SVG etc.

These are all markup languages.  The provisions must apply to a much wider range of web technologies. 
Suggested rewording:  • Ensure that requirements may be applied across technologies including markup and not markup, open and proprietary, web technologies.

> All success criteria will be mapped to the function requirements it aims to meet.

> Ensure that the conformance requirements are clear and testable (where possible) [Note: we will need language around areas that are not explicitly testable - or where subjective expert evaluation is required]

Testability is not optional or "where possible”.   
You cannot have a standard where it is not possible to determine if you conform or not.   
Any provision where you cannot determine clearly whether you meet the criterion OR not has to be advisory.    
This isnt a religious issue.  It is simply a requirement for ANY standard with conformance. 
It must be possible to determine if you have conformed. 

> Utilize the WCAG 2.0 A/AA/AAA structure.

> Success criteria need to be as broad as possible without becoming a 'catch-all' for any given requirement.

Good.  But the wording must be such that it CAN apply to all — or wording has to limit its application to what it should (can) apply to  

> Candidate success criteria will be peer reviewed and if too great in scope will be broken into more granular requirements.
> They must be testable.

Hmm  this conflicts with #2 above that says “where possible”.   This one is correct. 
I would add  (Automatic or human)   after testable. 

> They must be applicable to specific technologies.

I don’t think this is true.  So far all but one provision (4.1) are technology agnostic.   And many are completely technology independent. 

I would say that this should be the opposite.

They must not be technology specific. 

> They must always be true.

correct but confusingly worded.  it should be 

They should be worded as statements that are true or false. 
i.e. they are not  “ xxx must be true”  but rather    “xxx is yyy”     
for example   “all media includes captions”   
that way - if there is no media the statement is still true.
and we avoid the N/A problem. they always apply.  But are met 

> They are statements of 'what is'- when the statement of is true - then you have met the SC.

Yes this is right. 

>  
> Please feel free to add/comment etc. We are also interested in the 'institutional memory' available for those who are long-timers (lifers?) in the group. So we shall send out a survey soon to gather input.
>  
> Thanks
>  
> Josh
>  
> ------ Original Message ------
> From: "Katie Haritos-Shea" <ryladog@gmail.com <mailto:ryladog@gmail.com>>
> To: "White, Jason J" <jjwhite@ets.org <mailto:jjwhite@ets.org>>
> Cc: "Patrick Lauke" <redux@splintered.co.uk <mailto:redux@splintered.co.uk>>; "WCAG" <w3c-wai-gl@w3.org <mailto:w3c-wai-gl@w3.org>>
> Sent: 01/06/2016 10:25:34
> Subject: RE: acceptance criteria for new success criteria
>  
>> Andrew said...
>> "The term used during the development of WCAG 2.0 was “high inter-rater reliability”. I don’t recall our discussion of exactly what the requirements were, but my general recollection is that it entailed likely agreement by most reasonably informed evaluators (not the same as agreement by most “experts”, which, to my mind, is a lower standard that is easier to meet)."
>> 
>> I recall the “high inter-rater reliability” metric to be something like: if after testing content for a SC that 8 out of 10 experienced evaluators agreed....
>> 
>> 
>> Katie Haritos-Shea
>> 703-371-5545
>> 
>> On May 31, 2016 1:56 PM, "White, Jason J" <jjwhite@ets.org <mailto:jjwhite@ets.org>> wrote:
>> 
>> 
>> > -----Original Message-----
>> > From: Patrick H. Lauke [mailto:redux@splintered.co.uk <mailto:redux@splintered.co.uk>]
>> > On 31/05/2016 16:36, Detlev Fischer wrote:
>> > [...]
>> > > It is nice to believe that consensus is obtainable but even among
>> > > testers working to the same set of checkpoints with detailed rating
>> > > instructions, we frequently experience disagreement - mostly because
>> > > the issue context or the mapping of issues to SCs makes it hard to
>> > > agree on a fair rating.
>> >
>> > I'd like to wholeheartedly +1 Detlev's comment here.
>> 
>> The main strategy with which WCAG 2.0 seeks to address this problem is to encourage interpreters to consider the purpose of each requirement, not just its text, and to evaluate with the purpose in mind wherever there is ambiguity.
>> 
>> In technical standards, test suites and interoperability testing are a common technique for solving the problem, but they're of limited applicability here by reason of the many requirements that cannot be automatically evaluated.
>> 
>> A common legal approach to the problem of interpretation is to develop a case-law, i.e., normative decisions on specific interpretive questions that arise from concrete situations. This would work here (given a suitable group of experts and a good decision-making process), but the W3C isn't set up for it - the most that can be done is to provide non-normative guidance and to release a revised specification. To some extent, the techniques constitute interpretive material, though they are not normative and to that extent not authoritative.
>> 
>> Another possibility for a future version would be to have normative techniques as well as the general principles, guidelines and success criteria. Where the techniques apply, they are normative; where they don't apply, the more general standard has to be applied directly. This solution also has precedent in the way in which some legislative schemes are set up. Under this approach, the technology-specific guidance would be normative wherever it is applicable, but the general guidelines are also available to accommodate situations not encompassed by the specific requirements.
>> 
>> The main problem is that the techniques evolve too quickly for regulators, organizational policy setters and other parties who need to use the Guidelines; and there is a risk of losing the general principles amid all of the details, as well as the risk of encouraging more "legalistic" methods of interpretation that don't look to the broader purpose of making the Web more accessible but instead focus on the minutia of conformance and the language of normative statements. Also, if techniques were normative, the scrutiny that they would receive and the amount of work involved in bringing them to publication would likely increase substantially.
>> 
>> Before deciding among these and other solutions, we need to know how large the problem of interpretive disagreement is in WCAG 2 as it currently stands, then move the discussion into the planning process for preparing the next major version (not version 2.1, but the next significant revision).
>> 
>> 
>> ________________________________
>> 
>> This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
>> 
>> 
>> Thank you for your compliance.
>> 
>> ________________________________