Tests, Requirements, Evidence, and Methodologies

Dear Group,

During the previous ERT WG teleconference we identified an issue that needs to be resolved.

In the e-mail by Carlos Iglesias [1], he notes a discrepancy between the meaning and usage of the earl:TestRequirement class. The following is a rough summary of my understanding from the previous teleconference discussion, I hope it animates reactions and discussion:

Until recently we had an earl:TestCase class which we used to reference either a specific test (for example: "check if img-element has alt-attribute") *or* a requirement (for example: "WCAG 1.0 Checkpoint 1.1"). At the face-to-face we identified a confusion (especially during the presentation of the Test Case Description Language by FIT/BenToWeb) and so we decided to rename the class in order to better match both usages.

CarlosI argues that a test is always done for a requirement and so it does not make sense to reference the tests without the respective requirements. It seems that there is however a use case for the opposite: assertions about conformance to a requirement without being explicit about the test(s).

It seems to me that the evidence class may be an important part of this puzzle. If the earl:TestRequirement class truly references a requirement, the earl:Evidence could class be used to reference the specific tests that were carried out in order to make a conclusion about the result.

However, earl:Evidence call will reference other earl:Assertion classes. So at some point in the chain, we must have a direct reference to the specific test from an assertion. Besides, this may also be useful for other usages of EARL that may not be built around the requirement - test hierarchy.

This means we end up with the same problem we had in the first place: either we define that earl:TestRequirement (or whatever we call that class) to reference either a requirement *or* a test; or we resurrect earl:TestCase and define each of the classes to reference separate things.

Personally I favor the first option because it avoids confusion between the two classes. There may be also other options that have not been raised yet. I invite you to voice your opinions on the problem and possible solutions.

Finally, CarlosI also mentioned the evaluation methodology as another axis to this problem but we have not discussed how this fits in. Maybe it will become clearer in subsequent discussions.


Regards,
  Shadi
[1] <http://lists.w3.org/Archives/Public/public-wai-ert/2006Jan/0032>


-- 
Shadi Abou-Zahra     Web Accessibility Specialist for Europe | 
Chair & Staff Contact for the Evaluation and Repair Tools WG | 
World Wide Web Consortium (W3C)           http://www.w3.org/ | 
Web Accessibility Initiative (WAI),   http://www.w3.org/WAI/ | 
WAI-TIES Project,                http://www.w3.org/WAI/TIES/ | 
Evaluation and Repair Tools WG,    http://www.w3.org/WAI/ER/ | 
2004, Route des Lucioles - 06560,  Sophia-Antipolis - France | 
Voice: +33(0)4 92 38 50 64          Fax: +33(0)4 92 38 78 22 |

Received on Sunday, 22 January 2006 20:13:09 UTC