RE: Tests, Requirements, Evidence, and Methodologies

I completely agree Carlos. We need a Requirements class to reference
requirements and a TestCase class to reference test case.

The Evidence and Methodology classes have got me scratching my head.

I'm worried that we may be introducing new terminolgy. Certainly, 'evidence', is
new to me. I've always been of the opinion that the test case provides all the
evidence i need to prove a requirements success or failure. If i use my
imagination i can see that a Methodolgy class may come in handy but i'm having
trouble thinking of an instance where the methodolgy could not just as easily
be expressed as a test case.

Perhaps it would help if someone could provide a practical implementation to
demonstrate how they envision these classes to be used...

Kind regards,
David.

Quoting Carlos Iglesias <carlos.iglesias@fundacionctic.org>:

>
>
>
> Hi everbody,
>
> First of all, I will try to clarify again my understanding of the
> Requirements, Methodologies, Evidences and Tests mess
>
> IMO, there are two main parts in every Software engineering (and Web
> engineering) Test process (probably this applies to every engineering test
> process)
>
> Requirements - what you expect from the product
> Tests - the way you check whether you have what you expected or not
>
> If we focus on tests then we have Test Cases - a set of conditions under
> which a tester will determine if a requirement is satisfied
>
> E.g. WCAG 2.0 Test 49 - Content has valid language code
> [http://www.w3.org/WAI/GL/WCAG20/tests/test49.html]
>
> A Test Case typically includes
>
> * General info (id, creator, version, name, covered requirement,
> dependencies, etc.)
> * Process (environment, preconditions, postconditions, input data, expected
> results, etc.) and Actions step by step to be done to complete test case
> (what we call specific, individual or atomic tests?). E.g. (From the example
> above):
>
>    1. Determine the type of content within the content.
>    2. If the content is HTML, check the value of the html element's lang
> attribute.
>    3. If the content is XHTML 1.0, or any version of XHTML served as
> "text/html", check the values of both the html element's lang attribute and
> xml:lang attribute.
>    4. Note: both lang attributes must be set to the same value.
>    5. If the content is XHTML 1.1 or higher and served as type
> "application/xhtml+xml", check the value of the html element's xml:lang
> attribute.
>    6. Compare the language attribute value to valid language codes according
> to the ISO 639 specification.
>
> A Test Case never includes actual Test Results (this is the work of EARL)
>
> Finally there are Test Suites - a group or collection of test cases.
>
>
> Now let's see comments below.
>
>
> > Until recently we had an earl:TestCase class which we used to
> > reference either a specific test (for example: "check if
> > img-element has alt-attribute") *or* a requirement (for
> > example: "WCAG 1.0 Checkpoint 1.1"). At the face-to-face we
> > identified a confusion (especially during the presentation of
> > the Test Case Description Language by FIT/BenToWeb) and so we
> > decided to rename the class in order to better match both usages.
>
>
> In fact I think we were using TestRequirement and TestCase in a wrong way
> according to my understanding of them (as I explained above). We used both to
> reference the same thing:
>
> Sometimes specific tests (not test cases according to my understanding of
> them).
> Sometimes (mostly I think) requirements.
>
>
> > CarlosI argues that a test is always done for a requirement
> > and so it does not make sense to reference the tests without
> > the respective requirements. It seems that there is however a
> > use case for the opposite: assertions about conformance to a
> > requirement without being explicit about the test(s).
>
>
> The model could be something like:
>
> <earl:Assertion rdf:ID="#assertion">
>   <earl:assertedBy rdf:resource="#assertor"/>
>   <earl:subject rdf:resource="#subject"/>
>
>   <earl:requirement rdf:resource="#testrequirement"/>
>   <earl:testcase rdf:resource="#testcase"/>
>
>   <earl:result
> rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#pass"/>
> </earl:Assertion>
>
> earl:requirement --> required
>
> A pointer to the Requirement that is being tested
>
> earl:testcase --> optional
>
> A pointer to the Test Case (according to my understanding of them explained
> at the beginning of the message) that is used. It should provide general
> information (id, creator, version, etc.) and process information
> (preconditions, postconditions, input data... And specially, all the atomic
> test we use).
>
>
> > It seems to me that the evidence class may be an important
> > part of this puzzle. If the earl:TestRequirement class truly
> > references a requirement, the earl:Evidence could class be
> > used to reference the specific tests that were carried out in
> > order to make a conclusion about the result.
> >
> > However, earl:Evidence call will reference other
> > earl:Assertion classes. So at some point in the chain, we
> > must have a direct reference to the specific test from an
> > assertion. Besides, this may also be useful for other usages
> > of EARL that may not be built around the requirement - test hierarchy.
>
>
> Why not use an earl:TestRequirement class to truly reference a Requirement,
> an earl:TestCase class to truly reference a Test Case and an earl:Evidence
> class to truly reference an Evidence (if anybody knows what an evidence is
> ;o)?
>
>
> > This means we end up with the same problem we had in the
> > first place: either we define that earl:TestRequirement (or
> > whatever we call that class) to reference either a
> > requirement *or* a test; or we resurrect earl:TestCase and
> > define each of the classes to reference separate things.
> >
> > Personally I favor the first option because it avoids
> > confusion between the two classes. There may be also other
> > options that have not been raised yet. I invite you to voice
> > your opinions on the problem and possible solutions.
>
> Personally I think the first option doesn't avoid confusion at all, quite the
> contrary.
>
> If we agree that Test Requirements and Test Cases are clearly different
> things, then IMO it's obvious we must keep them separate. I can't imagine how
> calling requirement to something which is a test case, or vice versa, can
> avoid confusion. Let's try with a comparison:
>
> - We now that apples and lemons are different things (classes)
> - We decide to call all them apples (in theory to avoid confusion between
> apples and lemons)
> - I want to do an apple pie
> - I say "give me apples"
> - I receive apples and lemons but I didn't realise (How could I?)
> - My apple pie tastes really strange
> - Now I'm really confused and frustrated because I can't use the apples class
> to do an apple pie (Apparently the apples class doesn't contain only apples
> ¿¿¿???)
>
> > Finally, CarlosI also mentioned the evaluation methodology as
> > another axis to this problem but we have not discussed how
> > this fits in. Maybe it will become clearer in subsequent discussions.
>
>
> In fact I don't have a clear view of where Methodology and Evidence match in
> all this mess.
>
> Is Methodology something similar to my understanding of Test Case? Is it
> something wider? Is there any overlap between them?
>
> What in the hell is an Evidence? ;oD , could TestCase be a subclass of
> Evidence?
>
>
> > [1] <http://lists.w3.org/Archives/Public/public-wai-ert/2006Jan/0032>
>
>
> Regards,
>
> CI.
>
>
> --------------------------------------
>
> Carlos Iglesias
>
> Web Accessibility Unit - CTIC Foundation
> Science and Technology Park of Gijón
> 33203 - Gijón, Asturias, Spain
>
> phone: +34 984291212
> fax: +34 984390612
> email: carlos.iglesias@fundacionctic.org
> URL: http://www.fundacionctic.org
>
>

Received on Tuesday, 31 January 2006 12:39:00 UTC