RE: Re[2]: acceptance criteria for new success criteria

Have we lost sight of the fact that the “high inter reliability rating” in practice ("participants in the working group, by consensus, are confident that 8 out of 10 informed evaluators would agree in their application of the proposed success criterion") WAS and IS specific only to the Human Judgment testing of SC that require some level of it?

 

And I too, vote not to change that metric. 

 

Have you ever argues discussed a specific conformance issue with colleagues you respect but do not agree with? Da!  Of course!

 

The “General agreement, characterized by the absence of sustained opposition to substantial issues” makes sense for complete sense for how the Working Group reaches consensus. However, this topic specifically is about the testability of things/SC, or components of SC, that require human judgment to determine conformance. That is a different animal that a WG protocol for consensus.

 

​​​​​

 

 

 

* katie *

 

Katie Haritos-Shea 
Principal ICT Accessibility Architect (WCAG/Section 508/ADA/AODA)

 

Cell: 703-371-5545 |  <mailto:ryladog@gmail.com> ryladog@gmail.com | Oakton, VA |  <http://www.linkedin.com/in/katieharitosshea/> LinkedIn Profile | Office: 703-371-5545

 

From: Andrew Kirkpatrick [mailto:akirkpat@adobe.com] 
Sent: Thursday, June 2, 2016 11:08 AM
To: David MacDonald <david100@sympatico.ca>; josh@interaccess.ie
Cc: lisa.seeman <lisa.seeman@zoho.com>; White <jjwhite@ets.org>; Patrick H. Lauke <redux@splintered.co.uk>; w3c-wai-gl@w3.org
Subject: Re: Re[2]: acceptance criteria for new success criteria

 

David: I think the concept of high inter reliability of experts is as good as we can get. 

 

AWK: I think that the "General agreement, characterized by the absence of sustained opposition to substantial issues” actually matches the way that the group operated in practice, and continues to do so.  What we are really doing is following the WCAG and W3C process (see http://www.w3.org/WAI/GL/decision-policy), and our goal is unanimity, or an absence of objections.  Whether this means that everyone actively agrees or if most people agree and the rest can live with it doesn’t really matter.  

 

I’m concerned about the “8 of 10” since it starts to feel like voting and that hasn’t been the way that we have operated historically.

 

AWK

 

On Thu, Jun 2, 2016 at 9:51 AM, josh@interaccess.ie <mailto:josh@interaccess.ie>  <josh@interaccess.ie <mailto:josh@interaccess.ie> > wrote:

I think it sounds good, seems reasonable etc but as Patrick points out - not really practically testable in and of itself. It may also lend itself to the sway of the court of public opinion or received wisdom (albeit esteemed and informed) or worse  to 'group think' and/or lazy assumptions about what is best in a given situation.

 

Who will willingly play Cassandra in a court of her peers? [1]

 

[1] https://en.wikipedia.org/wiki/Cassandra 

 

------ Original Message ------

From: "lisa.seeman" <lisa.seeman@zoho.com <mailto:lisa.seeman@zoho.com> >

To: "White" <jjwhite@ets.org <mailto:jjwhite@ets.org> >

Cc: "Patrick H. Lauke" <redux@splintered.co.uk <mailto:redux@splintered.co.uk> >; "w3c-wai-gl@w3.org <mailto:w3c-wai-gl@w3.org> " <w3c-wai-gl@w3.org <mailto:w3c-wai-gl@w3.org> >

Sent: 02/06/2016 13:52:31

Subject: RE: acceptance criteria for new success criteria

 

I am happy with this standard  

 

 

"
In practice, the standard was: "participants in the working group, by consensus, are confident that 8 out of 10 informed evaluators would agree in their application of the proposed success criterion" (across a wide range of cases, I assume, though this last point is usually left implicit). "

 

All the best

Lisa Seeman

LinkedIn <http://il.linkedin.com/in/lisaseeman/> , Twitter <https://twitter.com/SeemanLisa> 



 


---- On Thu, 02 Jun 2016 15:43:59 +0300 White<jjwhite@ets.org <mailto:jjwhite@ets.org> > wrote ---- 



> -----Original Message----- 
> From: Patrick H. Lauke [mailto:redux@splintered.co.uk <mailto:redux@splintered.co.uk> ] 
> Sent: Thursday, June 2, 2016 4:57 AM 


> It may be a minor point, but: I'd prefer language that's a little less...specific. 
> Giving an actual figure of "8 out of 10" gives it a whiff of "it can be proven with 
> hard numbers", sure, but really: if there's ever a disagreement, do we really 
> expect somebody to gather 10 experts, get their opinions, and then make go for 
> the option that had 8 votes? What if it's 5 out of 10...a draw (which is probably 
> why you'd want 9 experts to be able to determine at least majority, barring 
> abstentions). 

In practice, the standard was: "participants in the working group, by consensus, are confident that 8 out of 10 informed evaluators would agree in their application of the proposed success criterion" (across a wide range of cases, I assume, though this last point is usually left implicit). 

So far as I am aware, no one has empirically tested the extent to which this standard is met by the success criteria that ultimately comprised WCAG 2.0. 


________________________________ 

This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. 


Thank you for your compliance. 

________________________________