Re: Tests, Requirements, Evidence, and Methodologies

Hi all,

Just thought i'd add my 2 cents to the discussion...

I'm definitely in agreeance with Carlos that the terminology of the test
requirement class is misleading. I think it should be renamed to Test Case. I
also agree with Carlos that there is a definite need for a test requirement
class that refers to an actual requirement. Test cases could then be mapped
against a requirement. In my opinion this is essential in order to obtain
meaningful test results. A requirement defines what it is your testing for, so
if it is not defined how do we know what were testing?

I'm still not %100 sure i understand the purpose of the proposed evidence
class.
Here is my interpretation...

The evidence class will be used to group the test cases that were performed in
order to determine whether a requirement has been met. I thin this would be
implemented something like this (this is just off the top of my head so
probably not accurate):

	<Requirement desc=WCAG 1.0 checkpoint 1.1>
		<Result>pass</Result>
		<Evidence>
			<TestCase>Check each image has an alt
attribute<Result>Pass</Result></TestCase>
			<TestCase>Check each frame has a longdesc
attribute<Result>Fail</Result></TestCase>
			<Reasoning>It was deemed that in this instance the purpose of the frames was
self evident and did not require a longdesc</Reasonong
		</Evidence>
	</Requirement>

This is the only way i could see a use for an evidence class. i.e. where a test
case fails but the requirement has been deemed to pass due a logical reason. Is
this roughly accurate?

Kind regards,
David.

> Quoting Shadi Abou-Zahra <shadi@w3.org>:
> 
> > 
> > Dear Group,
> > 
> > During the previous ERT WG teleconference we identified an issue that
> needs
> > to be resolved.
> > 
> > In the e-mail by Carlos Iglesias [1], he notes a discrepancy between the
> > meaning and usage of the earl:TestRequirement class. The following is a
> rough
> > summary of my understanding from the previous teleconference discussion, I
> > hope it animates reactions and discussion:
> > 
> > Until recently we had an earl:TestCase class which we used to reference
> > either a specific test (for example: "check if img-element has
> > alt-attribute") *or* a requirement (for example: "WCAG 1.0 Checkpoint
> 1.1").
> > At the face-to-face we identified a confusion (especially during the
> > presentation of the Test Case Description Language by FIT/BenToWeb) and so
> we
> > decided to rename the class in order to better match both usages.
> > 
> > CarlosI argues that a test is always done for a requirement and so it does
> > not make sense to reference the tests without the respective requirements.
> It
> > seems that there is however a use case for the opposite: assertions about
> > conformance to a requirement without being explicit about the test(s).
> > 
> > It seems to me that the evidence class may be an important part of this
> > puzzle. If the earl:TestRequirement class truly references a requirement,
> the
> > earl:Evidence could class be used to reference the specific tests that
> were
> > carried out in order to make a conclusion about the result.
> > 
> > However, earl:Evidence call will reference other earl:Assertion classes.
> So
> > at some point in the chain, we must have a direct reference to the
> specific
> > test from an assertion. Besides, this may also be useful for other usages
> of
> > EARL that may not be built around the requirement - test hierarchy.
> > 
> > This means we end up with the same problem we had in the first place:
> either
> > we define that earl:TestRequirement (or whatever we call that class) to
> > reference either a requirement *or* a test; or we resurrect earl:TestCase
> and
> > define each of the classes to reference separate things.
> > 
> > Personally I favor the first option because it avoids confusion between
> the
> > two classes. There may be also other options that have not been raised yet.
> I
> > invite you to voice your opinions on the problem and possible solutions.
> > 
> > Finally, CarlosI also mentioned the evaluation methodology as another axis
> to
> > this problem but we have not discussed how this fits in. Maybe it will
> become
> > clearer in subsequent discussions.
> > 
> > 
> > Regards,
> >   Shadi
> > [1] <http://lists.w3.org/Archives/Public/public-wai-ert/2006Jan/0032>
> > 
> > 
> > -- 
> > Shadi Abou-Zahra     Web Accessibility Specialist for Europe | 
> > Chair & Staff Contact for the Evaluation and Repair Tools WG | 
> > World Wide Web Consortium (W3C)           http://www.w3.org/ | 
> > Web Accessibility Initiative (WAI),   http://www.w3.org/WAI/ | 
> > WAI-TIES Project,                http://www.w3.org/WAI/TIES/ | 
> > Evaluation and Repair Tools WG,    http://www.w3.org/WAI/ER/ | 
> > 2004, Route des Lucioles - 06560,  Sophia-Antipolis - France | 
> > Voice: +33(0)4 92 38 50 64          Fax: +33(0)4 92 38 78 22 |

Received on Wednesday, 25 January 2006 12:43:33 UTC