Re: Another comment about confidence value.

On Sat, 16 Apr 2005 22:08:32 +0200, Nils Ulltveit-Moe <nils@u-moe.no>  
wrote:

> I promised myself to close the discussion on the conficence value last
> week. However during the workshop we have arranged at Agder University
> College last week on metamodel based accessibility testing, the issue
> about the confidence value came up again.
>
> The conclusion so far, is also that we do indeed need to be able to
> represent the confidence value of the test result.

Hi all,

well, I am glad the discussion has run on a bit. I have been convinced by  
it that there really is a good reason for having confidence, although I  
think it should be optional. I also think that it is important that  
confidence values state what ther scale is themselves.

This is the basis of my interoperability concern, and the reason I  
wondered if we wanted it at all. Whatever range of values we pick isn't  
nearly as important as the actual process used to assign a confidence  
rating to a particular result. So I would like to model the result as a  
blank node, and the confidence as a datatype.

So here is an example fragment assuming that the good folk at NILS publish  
their own confidence scheme for some tests (because they are proud of  
their name :-), and that Chris does for some others:

<earl:Assertion>
   <earl:message xml:lang="en">This is a fairly certain result for a  
machine that is acutally doing heuristic processing</earl:message>
   <earl:result r:parseType="Resource">
     <r:type r:resource="http://..earl..#fail" />
     <earl:confidence  
r:datatype="http://example.org/nils/datatypes/EARLtests">0,84</earl:confidence>
   </earl:result>
  ...
</earl:Assertion>

<earl:Assertion>
   <earl:message>Some other tools will have a much simpler range of  
possibilities for their confidence</earl:message>
   <earl:result r:parseType="Resource">
     <r:type r:resource="http://..earl..#pass" />
     <earl:confidence  
r:datatype="http://example.org/chris/datatypes/HML">Medium</earl:confidence>
   </earl:result>
   ...
</earl:Assertion>

<earl:Assertion>
   <earl:message xml:lang="en">Some tools, or people, are supremely  
confident. And sometimes they are also correct :-)</earl:message>
   <earl:result r:type r:resource="http://..earl..#pass" />
</earl:Assertion>

<earl:Assertion>
   <earl:message xml:lang="en">Having a fine-grained scheme for some  
particular system allows you to balance probabilities in multiple results.  
There are plenty of possibilities for cases where the confidence of a  
result either way is quite low but measurably different</earl:message>
   <earl:result r:parseType="Resource">
     <r:type r:resource="http://..earl..#fail" />
     <earl:confidence  
r:datatype="http://example.org/nils/datatypes/EARLtests">0,27</earl:confidence>
   </earl:result>
   <earl:result r:parseType="Resource">
     <r:type r:resource="http://..earl..#pass" />
     <earl:confidence  
r:datatype="http://example.org/nils/datatypes/EARLtests">0,31</earl:confidence>
   </earl:result>
</earl:Assertion>

I guess one of the things that we should assume is that any result without  
a confidence rating claims a confidence of 100% (which means you have to  
decide how you rate the assessor).

What do people think?

cheers

Chaals

-- 
Charles McCathieNevile                      Fundacion Sidar
charles@sidar.org   +61 409 134 136    http://www.sidar.org

Received on Sunday, 17 April 2005 09:59:35 UTC