Re: Confidences in accessibilty evaluation

Hi Shadi,

man, 18,.07.2005 kl. 12.42 +0200, skrev Shadi Abou-Zahra:
> Hi Nils,
> 
> Nils Ulltveit-Moe wrote:
> > However it also makes most sense to model barrier probabilities for each
> > disability group, especially when it comes to aggregation.
> 
> There is a clear difference here between barrier probabilities and confidence probabilities: while the first one addresses the effect of a violation on a given user, the second one addresses the occurence of a violation in the first place (i.e. 60% that there is a violation, this violation would cause X% barrier for person A).

I agree on that. It is two different (but somewhat related) measures.

> 
> > Maybe it still is too early to standardise the confidence value parameter?
> 
> If you actually mean barrier probabilities then yes, it may be too early for that. But the confidence property (as introduced by previous EARL schemas) has been actually deployed quite widely, however, not interoperably. I think it is high time to standardize the different approaches implemented by different developers.

The question is then how the confidence properties will be used? If they
are used as guidelines for expert evaluators on which parts of the
results from an automatic tool that needs further manual examination, as
is the case for conformance assessments, then different levels may make
sense. 

However, if the objective is some kind of automatic aggregation to
identify some kind of accessibility indicator, then different levels
will introduce more noise into the aggregated results, and is in general
not desireable.

Best regards,
Nils

> 
> Regards,
>   Shadi
> 
> 
> 
> > man, 18,.07.2005 kl. 11.07 +0200, skrev Shadi Abou-Zahra:
> > 
> >>Hi,
> >>
> >>Giorgio Brajnik wrote:
> >>
> >>>On 7/13/05, Nick Kew <nick@webthing.com> wrote:
> >>
> >>[SNIP]
> >>
> >>>>Were you on the call where we discussed this?  I think we were in 
> >>>>favour of allowing each tool its own choice over how to express 
> >>>>confidence values.
> >>
> >>[SNIP]
> >>
> >>>In the next months if I can I will do some work on this, which of
> >>>course will be shared with this group.
> >>
> >>We have talked about providing a more open mechanism for tools to provide their own methods of expressing confidence (especially for specialized domains or tools) but we also identified a need to roll out with a "built-in" default method that is unambiguous. As a first attempt, we agreed to look into the current High/Medium/Low values and see if we can define a robust model so that different tools would generate similar results.
> >>
> >>It seems to me that working with probabilities (or numbers in general) is a practical approach from the sense of aggregations and calculation. Also Nils has been previously posting some interesting work on this to the group. However, the major draw back is that this approach ties heavily with the test definitions or needs considerable benchmarking work to deliver the base values (probabilities).
> >>
> >>One approach may be to help develop the WCAG 2.0 test suites and then use them for benchmarking purposes. Tool developers could run their tools on such test suites and identify probability values on *checkpoint* level. Of course, developers could also have their own test suites that are public or proprietary if they (and their customers) prefer.
> >>
> >>Do you think that this would eventually work?
> >>
> >>Regards,
> >>  Shadi
> >>
> >>
> 
-- 
Nils Ulltveit-Moe <nils@u-moe.no>

Received on Monday, 18 July 2005 11:31:37 UTC