Re: EARL Guideline Pass/Fail Confidence

If the high confidence tells that a checkpoint was tested both with 
automatic test tools as well as by a human I would put that information to 
the description of the test case.

So we could have "partialtest1" and "partialtest2" for some automatic 
testing of checkpoint 1.1. In addition we could have a description of a 
combination test "trustedtest1" that refers to both automatic partial tests 
as well as the human test.

Separately, we could express trust for these partial and combination tests 
maybe only in cases when it was done by a trusted group of persons and not 
too long ago etc. This trust varies according to the individuals.

Some related things we have been experimenting with Annotea:
Nobu did some trust experimentations with Annotea shared bookmarks, where 
trusted people's bookmarks were used to change the order of the search 
results (will link some slides from 
http://www.w3.org/2001/Annotea/User/Papers.html after I get OK from Nobu).

Also Dom did some experimentations with marking messages as spam with spam 
annotations http://www.w3.org/2003/Talks/2003/Spamdemo/Overview.html. 
Similarly EARL Assertions or specific tests could be annotated (or 
bookmarked) as trustworthy or unreliable.

Marja

At 07:33 AM 1/26/2004 -0500, Charles McCathieNevile wrote:

>Hi,
>
>I don't think confidence is a particularly accurate measure. In some cases we
>are saying "cannotTell" but adding a suspicion, in some cases we are almost
>certain (some people seem always to be certain :-).
>
>I have no objection to people using a confidence scale, but I suspect that we
>should look at the use cases and whether we  can say something else more
>useful.
>
>(See also my recent email about the bug in my intro re using rdf:resource
>when it should be rdf:type or something even more complex...)
>
>Cheers
>
>Chaals
>
>On Thu, 15 Jan 2004, Chris Ridpath wrote:
>
> >
> >Charles has an example of EARL that shows how to express that a page
> >passes/fails an accessibility guideline. It's listed in his Coding EARL (for
> >non experts) document at:
> >http://www.w3.org/2001/sw/Europe/talks/200311-earl/all.htm
> >
> >The EARL code looks like:
> >
> >  <earl:Assertion>
> >    <earl:subject rdf:resource="#http://www.w3.org/" />
> >    <earl:result
> >rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#Pass"/>
> >    <earl:testcase
> >rdf:resource="http://example.org/1999/xhtml#transitional"/>
> >    <earl:assertedBy rdf:resource="http://validator.w3.org" />
> >    <earl:mode
> >rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#automatic"/>
> >    <earl:message>This page is valid XHTML</earl:message>
> >  </earl:Assertion>
> >
> >Would this be a better assertion if there was an added 'confidence'
> >statement? Example:
> >
> ><earl:confidence
> >rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#high" />
> >or
> ><earl:confidence rdf:resource=http://www.w3.org/WAI/ER/EARL/nmg-strawman#low
> >/>
> >
> >An automated checker tool can only detect some problems, not all. It's up to
> >a person to determine if the page passes all accessibility checks. For
> >example, only a person can determine if an image does/doesn't require a long
> >description.
> >
> >If the EARL expressed that the guideline was passed with a 'high' confidence
> >then it would mean that all accessibility checks had passed - machine and
> >human. If the confidence was 'low' then it would mean that only checks that
> >are machine testable had passed - one or more checks that require human
> >intervention had not passed.
> >
> >Using the confidence statement an automated checking tool could tell the
> >user that "likely the page will pass but you still need a human to make some
> >accessibility checks".
> >
> >Chris
> >
> >
> >
>
>Charles McCathieNevile  http://www.w3.org/People/Charles  tel: +61 409 134 136
>SWAD-E http://www.w3.org/2001/sw/Europe         fax(france): +33 4 92 38 78 22
>  Post:   21 Mitchell street, FOOTSCRAY Vic 3011, Australia    or
>  W3C, 2004 Route des Lucioles, 06902 Sophia Antipolis Cedex, France

Received on Wednesday, 28 January 2004 06:58:38 UTC