W3C home > Mailing lists > Public > w3c-wai-gl@w3.org > April to June 2005

[Techs] Definition of "Reliably human testable"

From: John M Slatin <john_slatin@austin.utexas.edu>
Date: Wed, 27 Apr 2005 13:54:09 -0500
Message-ID: <6EED8F7006A883459D4818686BCE3B3B7AE320@MAIL01.austin.utexas.edu>
To: <w3c-wai-gl@w3.org>
On the Techniques call today we discussed the proposed definition of the
term "reliably human testable":
 
<proposed>
[Definition: Reliably Human Testable: The technique can be tested by
human inspection and it is believed that at least 80% of knowledgeable
human evaluators
would agree on the conclusion. Tests done by people who understand the
guidelines should get the same results testing the same content for the
same success
criteria. The use of probabilistic machine algorithms may facilitate the
human testing process but this does not make it machine testable.]

</proposed>
 
Someone on the call asked whether the 80 percent figure represented an
arbitrary number.  I took an action item to find out and report back.
With terrific help from David Macdonald, I've got an answer:
 
The literature seems to support the 80 per cent figure.  In fact,
inter-rater reliability (percentage of agreement among multiple people
rating the same items) is considered "adequate," but 85% is considered
better. I think we're safe in using the 80 percent figure. If we go
lower than that it will be difficult to claim reliability.
 
John
 

"Good design is accessible design." 
John Slatin, Ph.D.
Director, Accessibility Institute
University of Texas at Austin
FAC 248C
1 University Station G9600
Austin, TX 78712
ph 512-495-4288, f 512-495-4524
email jslatin@mail.utexas.edu
web http://www.utexas.edu/research/accessibility/
<http://www.utexas.edu/research/accessibility/> 


 

 
Received on Wednesday, 27 April 2005 18:54:18 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 5 February 2014 23:39:37 UTC