Inter-rater reliability (was definition of testability)

Having suggested that we use the term "inter-rater reliability" and link
to a definition, I thought I'd better go hunt for some definitions.
Here's one that may complicate the issue a little:
 
<blockquote
cite=http://www.acvim.org/wwwfp/Diplomate/SAIM/InterRaterReliability.htm
>
Although some researchers use the terms "agreement" and "reliability" as
though they were interchangeable, there are some important differences
between the two terms:  "Interrater agreement is the extent of a match
between the raters as they assign scores to performance, behaviors, or
essays-whereas interrater reliability is the extent to which the raters
order the participants' performances, behaviors, or essays in the same
way" (Goodwin, 2001, p. 16).  
</blockquote>
Goodwin, L.D. (2001).  Interrater agreement and reliability.
Measurement in Physical Education and Exercise Science, 5, 13-34.
 
This difference was illustrated in a class I taught the other day on
writing for accessibility (before I had read the material quoted above).
I asked twenty students to read and rate three short texts on a scale
from 1 (poor) to 5 (excellent).  I did not provide criteria, part of my
point being that these folks, like other Web users, already *have*
criteria and apply them quickly and without much thought.  The twenty
particpants were divided into five groups.  There wasn't much
inter-rater "agreement" from group to group-- a given piece of text
received scores from 2-4, for example.  But there was substantial
inter-rater "reliability" according to the definition cited above: four
of the five groups *ranked* the three texts in the same order.
 
Hardly scientific, of course.
 
John


"Good design is accessible design." 
Please note our new name and URL!
John Slatin, Ph.D.
Director, Accessibility Institute
University of Texas at Austin
FAC 248C
1 University Station G9600
Austin, TX 78712
ph 512-495-4288, f 512-495-4524
email jslatin@mail.utexas.edu
web http://www.utexas.edu/research/accessibility/
<http://www.utexas.edu/research/accessibility/>