EARL: representing uncertainty

AIUI, EARL should facilitate the combination of evidences from
heterogenous sources.

It is in the nature of such evidence that it cannot always be
dealt with by a simple binary yes/no or pass/fail representation.
There will often be uncertainty.

Example: automated testing tool sees
  <img src="bullet.gif" width="8" height="8" alt="bullet.gif">

Being an intelligent tool, it can deduce that the alt text is
probably wrong.  But of course it can't be certain: the document
could be discussing the very subject of good and bad alt texts,
or have some such reason for the above.

Now if we modify the example just a little:
  <img src="bullet.gif" alt="bullet">

Now the tool can still question the construct, but the evidence
that it's wrong is weaker.  And if we change it again to
  <img src="bullet.gif" width="150" height="60" alt="bullet">

then it's more likely right than wrong.  A tool is unlikely to
attempt such a fine distinction, at least until someone makes a
PhD thesis of it :-)

If the purpose of a tool is simply to generate an instant report,
then that's fine: it simply flags the bullet for attention.
But if we're combining results from multiple sources, we want
to be able to represent uncertainty, and indeed relative
uncertainty (evidence A is more uncertain than evidence B).

If EARL is to be the basis for combining evidences, then I
would suggest that EARL should provide for the representation
of uncertainty.  In order to deal with relative uncertainty,
we should consider a numeric representation.  Some possibilities
are:
  * confidence intervals in [0,1].   This leaves it open for tools
    that have no interest in numeric measures to ignore them:
    any uncertain result could be implicitly assigned the
    entire [0,1] range.
  * Probability or Belief values.  Simple to work with, but
    can be problematic to assign in the first place.

I do not feel sufficiently confident with the notation to
discuss this in the context of sbp's EARL draft, and indeed
Sean has suggested in IRC that I might in any case be better
off presenting this in its own right.  So, how do others
feel about this?

-- 
Nick Kew

Site Valet - the essential service for anyone with a website.
<URL:http://valet.webthing.com/>

Received on Tuesday, 6 November 2001 19:36:11 UTC