RE: Assessment outcomes

The tool reports a cannotTell, mode automatic.

The person reports a pass (or fail), mode manual.

You can select based on the mode (if you generally trust people over tools)
or you can select based on a particular assertor if you have more detailed
information.

For example, my WCAG evaluating tool requires a person to test something and
make an assertion. It will soon (I hope :-) put its results onto an annotea
system, as will other automatic tests.

What to do with apparently conflicting results is a big question anyway. What
if a tool is certain about but something, but a smart person says tehy cannot
tell? Normally I know which I would trust, but knowing who said what is
important for me...

cheers

Chaals

On Fri, 1 Aug 2003, Shadi Abou-Zahra wrote:

>
>how would you then really describe a pass or fail (by manual review)
>after an initial cannotTell returned by a tool?
>
>
>> -----Original Message-----
>> From: w3c-wai-er-ig-request@w3.org
>> [mailto:w3c-wai-er-ig-request@w3.org] On Behalf Of Charles
>> McCathieNevile
>> Sent: Freitag, 01. August 2003 01:07
>> To: Nick Kew
>> Cc: w3c-wai-er-ig@w3.org
>> Subject: Re: Assessment outcomes
>>
>>
>>
>> I'd do this differently. The idea is that you make a property called
>> nick:accept - it is generally a subclass of earl:fail,
>> because the implied
>> result is that the checkpoint has not been met, but in your
>> reporting you
>> treat it as you treat an earl:pass.
>>
>> This leaves you with earl:cannotTell for your proposed
>> Review, and earl:fail
>> for your proposed repair property. And declaring the schema
>> like that means
>> that your results are readily interoperable with others -
>> nick:acceptIt might
>> be considered fine in the reporting logic for some checkpoints.
>>
>> This strikes me as similar to Hixie's desire to subclass fail
>> according to
>> whether it was a recorded bug that caused the problem, or
>> consequent on
>> something else, or a catastrophic failure, etc...
>>
>> my 2 bits
>>
>> Chaals
>>
>> On Thu, 31 Jul 2003, Nick Kew wrote:
>>
>> >
>> >
>> >Someone mentioning EARL prompts me to post this ...
>> >
>> >In the course of developing Site Valet Enterprise Edition 2.0,
>> >I've integrated a full accessibility audit trail.  Pages will be
>> >assessed by an automatic agent, and may (or must, according to
>> >local policy) then be reassessed by a human.
>> >
>> >For the AccessValet desktop tool, it is sufficient to generate
>> >a result (Pass/Fail/Unknown/unchecked) and a separate conclusion.
>> >But for the sitewide database and audit trail, what is required
>> >is a single-word status to appear in query results, etc.  I'm
>> >currently using a slightly different vocabulary, that doesn't
>> >fit as well as (IMO) it should with EARL:
>> >
>> > * Pass
>> >	No problems with that one
>> > * Accept
>> >	An informed decision not to comply with some part of the
>> >	guidelines.
>> > * Review
>> >	No conclusion has been reached and the page should be reviewed.
>> > * Repair
>> >	The page has been reviewed and repairs have been identified.
>> >	A much more positive thing than "Fail" to say to users!
>> >
>> >These are supported by more detailed reports equivalent to the
>> >Executive Summary from the desktop product.  But should I be
>> concerned
>> >about departing from EARL vocabulary in the above?
>> >
>> >
>> >
>> >--
>> >Nick Kew
>> >
>>
>> Charles McCathieNevile  http://www.w3.org/People/Charles
>> tel: +61 409 134 136
>> SWAD-E http://www.w3.org/2001/sw/Europe         fax(france):
>> +33 4 92 38 78 22
>>  Post:   21 Mitchell street, FOOTSCRAY Vic 3011, Australia    or
>>  W3C, 2004 Route des Lucioles, 06902 Sophia Antipolis Cedex, France
>>
>

Charles McCathieNevile  http://www.w3.org/People/Charles  tel: +61 409 134 136
SWAD-E http://www.w3.org/2001/sw/Europe         fax(france): +33 4 92 38 78 22
 Post:   21 Mitchell street, FOOTSCRAY Vic 3011, Australia    or
 W3C, 2004 Route des Lucioles, 06902 Sophia Antipolis Cedex, France

Received on Thursday, 31 July 2003 19:23:33 UTC