Re: Acceptance Criteria for proposals for new Success Criteria

Ah, I see.  I don’t think that will work.  Testability is an absolute requirement.  If an SC isn’t testable then it won’t become an SC, as it was with WCAG 2.0.  Worth noting that there is a difference between “testable” and “machine testable” and this covers both.
[Jason] Being reliably testable is not a straightforward property that an SC either possesses or lacks. This is especially true of human-testable requirements, which can be made more or less reliably testable (i.e., with more or less probability of agreement among informed reviewers).
There’s an interesting paper which provides evidence and analysis to suggest that we were not as effective as we hoped we would be in defining reliably human testable success criteria – see the reference below. In particular, in a study of a sample of Web pages, 80% agreement even among informed reviewers with accessibility expertise was not achieved; yet this was once the degree of agreement that we sought to reach in defining reliably human testable SCs.
I think we should strive for more reliable testing, but not at the expense of providing general and useful requirements that solve accessibility problems. There is also a need for more research into ways of improving the extent to which success criteria are reliably testable (especially in the next generation of WAI Guidelines).

The reference is:

Brajnik, G., Yesilada, Y., & Harper, S. (2012). Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0. ACM Transactions on Accessible Computing (TACCESS), 4(2), 8.

You’ve added “reliably” to the statement about testability.  I believe that adding reliably to this changes the dependance from being just on the potential testability of the SC as understood by the Working Group (who should understand the content and the intent better than anyone) into a measure that is also affected by the clarity of the language and how simple and straightforward the concept covered by the SC is.

In the study, they cite items like 2.4.7 (Focus Visible) as on 33-51% reliably tested, even by experienced evaluators.  I would say that whether a keyboard focus indicator is visible is 99+% testable. Similarly the study says that 1.3.2 (meaningful sequence) is 40-50% reliably testable.  I would say that it is 99+% testable, but it is important that people understand how to do this, and that is where the gaps begin.  Some people will try to read a page with a screen reader but don’t really know how, others will inspect the DOM order, but there are correct answers to both of these.

Explaining how to test the SC, so it may be done reliably (and don’t get me wrong, this is a desirable outcome) is something that we need to make sure that we keep in mind for techniques and understanding documents, and also in making sure that the SC itself is crisp and clear, but I don’t think that we can make reliable testability a measured goal for SC.

The bar being set here is testable – if an SC isn’t testable then it certainly can’t be reliably testable.


6.Apply to all content, unless specific exceptions are included in the success criteria (e.g. "except interruptions involving an emergency").
not sure what I’m missing.  The original text says that if there are exceptions that they needs to be included in the SC.  That is explicitly mentioning them, isn’t it?
[Jason] My point is that exceptions and preconditions often aren’t the same. Stating that a requirement applies in some well defined circumstances (but not in most cases) isn’t an example of providing an exception to a general rule. “Preconditions for the application of success criteria should be stated explicitly” is the requirement, I think.

OK, how about:
(SC’s shall) Apply to all content unless preconditions for the application of the success criteria are explicitly identified (e.g. "except interruptions involving an emergency”).

AWK