- From: Christophe Strobbe <christophe.strobbe@esat.kuleuven.be>
- Date: Wed, 12 Oct 2005 14:31:26 +0200
- To: public-wai-ert@w3.org
Hi Shadi, At 16:00 11/10/2005, Shadi Abou-Zahra wrote: >Hi Christophe, > >Christophe Strobbe wrote: >>>* What is the impact and severity of inconsistency regarding the location? >> >>One obvious consequence is that it becomes hard to compare and benchmark >>evaluation tools, but that is admittedly a very specific use case of EARL. > >One could still compare if tools produce the same results as well as the >number of instances of that result per test-subject (count the number of >location properties). You're right. >Is it really essential to compare the specific locations that actually >triggered the results as well or is the location more suitable for human >consumption? I think it would make comparisons more reliable. Imagine that there are six instances of a similar construct in a page: - tool A says that instances 1, 2 and 4 pass, and that instances 3, 5 and 6 fail CP 2.2 (colour contrast), - tool B says that instances 1, 2 and 3 pass, and that instances 4, 5 and 6 fail CP 2.2, so the number of instances is the same in each report and only the locations can tell you if both tools find the same problems. But this case is only relevant to makers of test suites for benchmarks and can probably be ignored in evaluations of real websites. Regards, Christophe Strobbe >Regards, > Shadi -- Christophe Strobbe K.U.Leuven - Departement of Electrical Engineering - Research Group on Document Architectures Kasteelpark Arenberg 10 - 3001 Leuven-Heverlee - BELGIUM tel: +32 16 32 85 51 http://www.docarch.be/ Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
Received on Wednesday, 12 October 2005 12:32:51 UTC