RE: Revised statement on testability (was" Definition of human te stability)

I continue to be concerned about having a definition for testability that is
based solely on the belief of the members of the working group and is not
backed up by empirical evidence. We collectively have a good deal of
expertise and certainly can make educated guesses about a guideline's
testability, or the inter-rater reliability of a human testing process. But,
unless we test those guesses, we will probably be surprised by one or two of
them after the guidelines are published. It's always the case that one
discovers something you didn't think of no matter how hard one has
considered.
 
Therefore I think we need a requirement for ourselves that our testability
assertions themselves be tested prior to finalization of the guidelines. We
need to obtain comprehensive sets of test cases for each guideline, subject
them to human testing, and measure the inter-rater reliability. If the
measurement meets a threshold we set, we define the guideline as testable,
otherwise we define it as untestable and take action from there (e.g.,
change or remove the guideline).
 
If we undertake this process we could change the last sentence of the
definition from "We believe that different people who understand the
guidelines will usually get the same or very similar results when they test
the same success criterion" to "We have determined by investigation
that...".
 
This creates a huge new task and exit criterion for us. But I think it is
essential if we have a requirement that all guidelines be testable. If I
were an Advisory Committee representative I would not vote in favour of a
specification that included a major conformance criterion that had not been
empirically verified to be possible, so I have to imagine this is a hurdle
we will face later in the process.
 
Michael
 

-----Original Message-----
From: John M Slatin [mailto:john_slatin@austin.utexas.edu]
Sent: Tuesday, May 11, 2004 4:02 PM
To: w3c-wai-gl@w3.org
Subject: Revised statement on testability (was" Definition of human
testability)


Hello,
 
Yvette and I have been working on the statement about human testability.
What we've come up with isn't so much a definition as a statement of what we
think the group believes-- or is at least a statement that we would like to
be able to make by the time WCAG 2.0 reaches Candidate Recommendation.  This
statement would appear in the Conformance section along with the explanation
of how we sorted success criteria into the various levels.
 
<begin proposed>
The Working Group believes that tests can show whether Web content passes or
fails each success criterion. Tests can be done by computer programs or by
people who understand these guidelines. We believe that different people who
understand the guidelines will usually get the same or very similar results
when they test the same success criterion.
</end proposed>
 

John
"Good design is accessible design." 
Please note our new name and URL!
John Slatin, Ph.D.
Director, Accessibility Institute
University of Texas at Austin
FAC 248C
1 University Station G9600
Austin, TX 78712
ph 512-495-4288, f 512-495-4524
email jslatin@mail.utexas.edu
web  <http://www.utexas.edu/research/accessibility/>
http://www.utexas.edu/research/accessibility/