Re: [Techs] Definition of "Reliably human testable"

The "test environment" for the human evaluators may be important.  If the 
testing is done with human evaluators "in isolation", as opposed to being 
in "focus groups", undue influence among evaluators may be avoided, and the 80% 
figure may be more credible.. (maybe this factor has been considered in 
the "literature"?) 

  Quoting John M Slatin <john_slatin@austin.utexas.edu>:

> On the Techniques call today we discussed the proposed definition of the
> term "reliably human testable":
>  
> <proposed>
> [Definition: Reliably Human Testable: The technique can be tested by
> human inspection and it is believed that at least 80% of knowledgeable
> human evaluators
> would agree on the conclusion. Tests done by people who understand the
> guidelines should get the same results testing the same content for the
> same success
> criteria. The use of probabilistic machine algorithms may facilitate the
> human testing process but this does not make it machine testable.]
> 
> </proposed>
>  
> Someone on the call asked whether the 80 percent figure represented an
> arbitrary number.  I took an action item to find out and report back.
> With terrific help from David Macdonald, I've got an answer:
>  
> The literature seems to support the 80 per cent figure.  In fact,
> inter-rater reliability (percentage of agreement among multiple people
> rating the same items) is considered "adequate," but 85% is considered
> better. I think we're safe in using the 80 percent figure. If we go
> lower than that it will be difficult to claim reliability.
>  
> John
>  
> 
> "Good design is accessible design." 
> John Slatin, Ph.D.
> Director, Accessibility Institute
> University of Texas at Austin
> FAC 248C
> 1 University Station G9600
> Austin, TX 78712
> ph 512-495-4288, f 512-495-4524
> email jslatin@mail.utexas.edu
> web http://www.utexas.edu/research/accessibility/
> <http://www.utexas.edu/research/accessibility/> 
> 
> 
>  
> 
>  
>

Received on Thursday, 28 April 2005 13:19:53 UTC