Re: Minutes for Teleconference on 24 May, 2005

Hi Shadi,

I saw the discussion on confidence values came up again. Here is some
ideas I am working with for aggregating the confidence level parameters.
I hope it can be useful in this discussion. It also explains why it may
be helpful to convey confidence level parameters modelled as
probabilities for those tools that can do that.

We are looking into modelling the confidence level parameter as the
probability that a given WCAG checkpoint fail will cause an
accessibility barrier for a given user. 

We assume that average probabilities somehow can be found, maybe by
user-testing. We also assume initially that the test results are
independent of each other. A negative test Tneg is here a test that
indicates that there is a barrier with some probability, and a positive
test Tpos indicates that this page with some probability is accessible.

With these definitions, we can calculate the probability p(Tneg) that
all of these negative WCAG tests Tneg together does not cause an
accessibility barrier as:

p(not_barrier)=(1-p(Tneg1))(1-p(Tneg2)...(1-p(Tnegn))

Tneg1,Tneg2,...,Tnegn are EARL negative test results (the same or
different tests).

The probability that all these tests cause a barrier is then:

p(barrier)=1-(1-p(Tneg1))(1-p(Tneg2)...(1-p(Tnegn))

We can extend this model to include the case of positive tests that also
have a confidence value by multiplying the confidence value that a
positive test is indeed a pass:

p(barrier)=1-(1-p(Tneg1))(1-p(Tneg2)...(1-p(Tnegn))p(Tpos1)p(Tpos2),...,p(Tposn)

Where Tpos1,Tpos2,...,Tposn are EARL positive tests

For example: Consider a test Tneg1 for WCAG checkpoint 1.1 (alt text)
that fails, and that in general, missing alt-text causes an
accessibility barrier in 80% of the cases. In addition, a test Tneg2 for
WCAG checkpoint 10.1 fails, because popup windows are used on the tested
page. The probability that this test causes a barrier for a blind person
is 30% (blind persons have problems with both of these checkpoints).
The probability that these two problems causes an accessibility barrier,
is then:

p(barrier)=1-(1-0.8)(1-0.3)=0.86

Assume then that the previous example also has a third test, that is a
positive WCAG 1.1 test that passes with confidence 90%. This gives a
probability for a barrier of:

p(barrier)=1-(1-0.8)(1-0.3)0.9=0.874

I.e. even a positive test that sometimes will cause a barrier will
increase the expected probability of a barrier. Positive tests that have
a confidence of 100% will not influence the end result, and negative
tests of 100% will cause a barrier with probability 1, as expected.

Another advantage of this, is that the results will give a quite robust
indication of whether there is a real accessibility barrier or not for a
given user group, website or web page, depending on how it was tested.
It should be less influenced by implementation differences in the
various accessibility tools, since the calculation of a barrier
probability is the product of several probability factors. If one is
missing, the confidence level will be different, but the end result may
still not change significantly.

Please note that accessibility assessment can probably not be considered
as an ideal stochastic system, so the assumption that the tests are
unrelated may not be true.

The impact of a dependency between the parameters would mean that 
p(a|b) = p(a intersect b)/p(b). p(a intersect b) is the intersection
(dependency) between a and b. In the worst case scenario, p(a)=p(b) (it
is the same event, they are totally dependant.) This would cause us to
multiply p(a) one time too much, so that in our model we would calculate
a higher probability for a barrier than there really is.

Anyway, one nice property of the simple model, is that the calculation
is consistently moving towards more probability for a barrier the more
hurdles there are in the document or web site. It may converge too fast
towards a barrier, depending on possible dependencies between the WCAG
checkpoints. 

What are your comments on this?

Best regards,
-- 
Nils Ulltveit-Moe <nils@u-moe.no>

tir, 24,.05.2005 kl. 19.27 +0200, skrev Shadi Abou-Zahra:
> ERT WG,
> 
> Please find the minutes for the teleconference on 24 May, 2005:
>  <http://www.w3.org/2005/05/24-er-minutes.html>
> 
> Next meeting is next Tuesday 31 May, 2005.
> 
> Regards,
>  Shadi
> 
> 
-- 
Nils Ulltveit-Moe <nils@u-moe.no>

Received on Tuesday, 24 May 2005 19:42:34 UTC