Re: Confidences in accessibilty evaluation

Hi Shadi and Giorgio,

It is hard to see a general way of dealing with the confidence values.
The problem with choosing fixed levels (low, medium, high), is that they
are not aggregatable. For us, that is very important. We are looking
into possibilities of modelling and aggregating accessibility barrier
probabilities, and to do this, the probabilities should be modelled as
real positive numbers. 

However it also makes most sense to model barrier probabilities for each
disability group, especially when it comes to aggregation. That would
imply that not only one, but in some cases a list of barrier probability
values would be needed if the tools are going to provide the barrier
probabilities for each disability group. This would probably be too
specialised for EARL, since EARL is supposed to be a general evaluation
and repair language, and not specific for some web accessibility model.

It is indeed a very complex problem. Good use cases and requirements for
the confidence values that will fit into an overall model of how to
aggregate values over a page, use scenario or similar will be needed
before one can decide on if and how to deal with the confidence values
in EARL. Maybe it still is too early to standardise the confidence value
parameter?

Best regards,
Nils Ulltveit-Moe

man, 18,.07.2005 kl. 11.07 +0200, skrev Shadi Abou-Zahra:
> Hi,
> 
> Giorgio Brajnik wrote:
> > On 7/13/05, Nick Kew <nick@webthing.com> wrote:
> [SNIP]
> >>Were you on the call where we discussed this?  I think we were in 
> >>favour of allowing each tool its own choice over how to express 
> >>confidence values.
> [SNIP]
> > In the next months if I can I will do some work on this, which of
> > course will be shared with this group.
> 
> We have talked about providing a more open mechanism for tools to provide their own methods of expressing confidence (especially for specialized domains or tools) but we also identified a need to roll out with a "built-in" default method that is unambiguous. As a first attempt, we agreed to look into the current High/Medium/Low values and see if we can define a robust model so that different tools would generate similar results.
> 
> It seems to me that working with probabilities (or numbers in general) is a practical approach from the sense of aggregations and calculation. Also Nils has been previously posting some interesting work on this to the group. However, the major draw back is that this approach ties heavily with the test definitions or needs considerable benchmarking work to deliver the base values (probabilities).
> 
> One approach may be to help develop the WCAG 2.0 test suites and then use them for benchmarking purposes. Tool developers could run their tools on such test suites and identify probability values on *checkpoint* level. Of course, developers could also have their own test suites that are public or proprietary if they (and their customers) prefer.
> 
> Do you think that this would eventually work?
> 
> Regards,
>   Shadi
> 
> 
-- 
Nils Ulltveit-Moe <nils@u-moe.no>

Received on Monday, 18 July 2005 10:11:13 UTC