From: Nils Ulltveit-Moe <nils@u-moe.no>

Date: Tue, 24 May 2005 21:49:23 +0200

To: shadi@w3.org

Cc: public-wai-ert@w3.org

Message-Id: <1116964163.12864.181.camel@moe-ulltveit-moe.com>

Date: Tue, 24 May 2005 21:49:23 +0200

To: shadi@w3.org

Cc: public-wai-ert@w3.org

Message-Id: <1116964163.12864.181.camel@moe-ulltveit-moe.com>

Hi Shadi, I saw the discussion on confidence values came up again. Here is some ideas I am working with for aggregating the confidence level parameters. I hope it can be useful in this discussion. It also explains why it may be helpful to convey confidence level parameters modelled as probabilities for those tools that can do that. We are looking into modelling the confidence level parameter as the probability that a given WCAG checkpoint fail will cause an accessibility barrier for a given user. We assume that average probabilities somehow can be found, maybe by user-testing. We also assume initially that the test results are independent of each other. A negative test Tneg is here a test that indicates that there is a barrier with some probability, and a positive test Tpos indicates that this page with some probability is accessible. With these definitions, we can calculate the probability p(Tneg) that all of these negative WCAG tests Tneg together does not cause an accessibility barrier as: p(not_barrier)=(1-p(Tneg1))(1-p(Tneg2)...(1-p(Tnegn)) Tneg1,Tneg2,...,Tnegn are EARL negative test results (the same or different tests). The probability that all these tests cause a barrier is then: p(barrier)=1-(1-p(Tneg1))(1-p(Tneg2)...(1-p(Tnegn)) We can extend this model to include the case of positive tests that also have a confidence value by multiplying the confidence value that a positive test is indeed a pass: p(barrier)=1-(1-p(Tneg1))(1-p(Tneg2)...(1-p(Tnegn))p(Tpos1)p(Tpos2),...,p(Tposn) Where Tpos1,Tpos2,...,Tposn are EARL positive tests For example: Consider a test Tneg1 for WCAG checkpoint 1.1 (alt text) that fails, and that in general, missing alt-text causes an accessibility barrier in 80% of the cases. In addition, a test Tneg2 for WCAG checkpoint 10.1 fails, because popup windows are used on the tested page. The probability that this test causes a barrier for a blind person is 30% (blind persons have problems with both of these checkpoints). The probability that these two problems causes an accessibility barrier, is then: p(barrier)=1-(1-0.8)(1-0.3)=0.86 Assume then that the previous example also has a third test, that is a positive WCAG 1.1 test that passes with confidence 90%. This gives a probability for a barrier of: p(barrier)=1-(1-0.8)(1-0.3)0.9=0.874 I.e. even a positive test that sometimes will cause a barrier will increase the expected probability of a barrier. Positive tests that have a confidence of 100% will not influence the end result, and negative tests of 100% will cause a barrier with probability 1, as expected. Another advantage of this, is that the results will give a quite robust indication of whether there is a real accessibility barrier or not for a given user group, website or web page, depending on how it was tested. It should be less influenced by implementation differences in the various accessibility tools, since the calculation of a barrier probability is the product of several probability factors. If one is missing, the confidence level will be different, but the end result may still not change significantly. Please note that accessibility assessment can probably not be considered as an ideal stochastic system, so the assumption that the tests are unrelated may not be true. The impact of a dependency between the parameters would mean that p(a|b) = p(a intersect b)/p(b). p(a intersect b) is the intersection (dependency) between a and b. In the worst case scenario, p(a)=p(b) (it is the same event, they are totally dependant.) This would cause us to multiply p(a) one time too much, so that in our model we would calculate a higher probability for a barrier than there really is. Anyway, one nice property of the simple model, is that the calculation is consistently moving towards more probability for a barrier the more hurdles there are in the document or web site. It may converge too fast towards a barrier, depending on possible dependencies between the WCAG checkpoints. What are your comments on this? Best regards, -- Nils Ulltveit-Moe <nils@u-moe.no> tir, 24,.05.2005 kl. 19.27 +0200, skrev Shadi Abou-Zahra: > ERT WG, > > Please find the minutes for the teleconference on 24 May, 2005: > <http://www.w3.org/2005/05/24-er-minutes.html> > > Next meeting is next Tuesday 31 May, 2005. > > Regards, > Shadi > > -- Nils Ulltveit-Moe <nils@u-moe.no>Received on Tuesday, 24 May 2005 19:42:34 GMT

*
This archive was generated by hypermail 2.2.0+W3C-0.50
: Tuesday, 8 January 2008 14:18:25 GMT
*