[techs] testability (was Re: Issues raised from Techniques teleconference)

Michael Cooper writes:
 > 
 > One proposed requirement is that Techniques state whether they are testable;
 > that has been further refined to be that they should state whether they are
 > a) machine testable, b) not machine testable but human testable, or c) not
 > testable. "Machine testable" means that a machine can validate conformance
 > to the Technique without human input. We include the possibility of "not
 > testable" Techniques for those that apply to Additional Ideas in a WCAG
 > Checkpoint and are not required to be testable.
 > 
 > In discussion we identified a couple issues with this. First, the line
 > between machine testable and not is fuzzy. As evaluation tools improve a
 > Technique that was not machine testable may become so; therefore it may not
 > be desirable to hard-code the testability nature.

Here is a proposal. A technique is machine testable if and only if
there is known to be an algorithm that will determine, with complete
reliability, whether the technique has been implemented or not. This
is intended to exclude probabilistic approaches, which by their nature
involve some possibility of error. Techniques that are not machine
testable are human testable, but there may exist probabilistic or
other techniques for assisting the human being in carrying out the
testing by identifying a large class of possible cases of
non-implementation.

My focus on algorithms is intended to take advantage of the fact that
tools change much more quickly than algorithms do, and if a
knowledgeable person can't identify an algorithm that will yield the
correct result reliably, then the technique isn't machine testable in
the sense that Michael characterized above (of not requiring human
input) until some advance has been made not just in evaluation tools,
but in the state of the art of, say, artificial intelligence or some
other field of research.

Second, we could not agree
 > that Techniques could ever be truly untestable; if that were the case, there
 > would be no measurable benefit if a document includes features guided by
 > Additional Ideas. Although we did not propose removing the requirement to
 > provide information on testability, we think that testability is unclear and
 > needs further discussion.

The working group has already agreed that a success criterion, and
this applies to the techniques also, is deemed untestable if it is
considered that 8 out of 10 informed evaluators wouldn't agree in
their judgments in a
very wide variety of cases. The term "non-testable" may be misleading;
it doesn't mean that no possible evidence can count for or against the
claim that a technique has been implemented, it just means that in the
opinion of the working group the available (human) testing methods
don't meet the required threshold of reliability.

In practice, any technique related to an item appearing under a WCAG
review requirement (including those under the "additional ideas"
sections of the current document) won't be testable in the relevant
sense, though as Lisa pointed out in connection with checkpoint 4.1
there may exist tools that can facilitate a human evaluator by
applying suitable heuristics.

The above proposals are based as far as possible on consensus
decisions of the working group with respect to the notion of
testability. I have tried to provide clear and workable definitions
that address the issues identified.

Received on Saturday, 11 January 2003 00:16:58 UTC