RE: Tests, Requirements, Evidence, and Methodologies

 

Hi everbody,

First of all, I will try to clarify again my understanding of the Requirements, Methodologies, Evidences and Tests mess

IMO, there are two main parts in every Software engineering (and Web engineering) Test process (probably this applies to every engineering test process)

Requirements - what you expect from the product
Tests - the way you check whether you have what you expected or not

If we focus on tests then we have Test Cases - a set of conditions under which a tester will determine if a requirement is satisfied

E.g. WCAG 2.0 Test 49 - Content has valid language code [http://www.w3.org/WAI/GL/WCAG20/tests/test49.html]

A Test Case typically includes

* General info (id, creator, version, name, covered requirement, dependencies, etc.)
* Process (environment, preconditions, postconditions, input data, expected results, etc.) and Actions step by step to be done to complete test case (what we call specific, individual or atomic tests?). E.g. (From the example above):

   1. Determine the type of content within the content.
   2. If the content is HTML, check the value of the html element's lang attribute.
   3. If the content is XHTML 1.0, or any version of XHTML served as "text/html", check the values of both the html element's lang attribute and xml:lang attribute.
   4. Note: both lang attributes must be set to the same value.
   5. If the content is XHTML 1.1 or higher and served as type "application/xhtml+xml", check the value of the html element's xml:lang attribute.
   6. Compare the language attribute value to valid language codes according to the ISO 639 specification.

A Test Case never includes actual Test Results (this is the work of EARL)

Finally there are Test Suites - a group or collection of test cases.


Now let's see comments below.


> Until recently we had an earl:TestCase class which we used to 
> reference either a specific test (for example: "check if 
> img-element has alt-attribute") *or* a requirement (for 
> example: "WCAG 1.0 Checkpoint 1.1"). At the face-to-face we 
> identified a confusion (especially during the presentation of 
> the Test Case Description Language by FIT/BenToWeb) and so we 
> decided to rename the class in order to better match both usages.


In fact I think we were using TestRequirement and TestCase in a wrong way according to my understanding of them (as I explained above). We used both to reference the same thing:

Sometimes specific tests (not test cases according to my understanding of them).
Sometimes (mostly I think) requirements.


> CarlosI argues that a test is always done for a requirement 
> and so it does not make sense to reference the tests without 
> the respective requirements. It seems that there is however a 
> use case for the opposite: assertions about conformance to a 
> requirement without being explicit about the test(s).


The model could be something like:

<earl:Assertion rdf:ID="#assertion">
  <earl:assertedBy rdf:resource="#assertor"/>
  <earl:subject rdf:resource="#subject"/>

  <earl:requirement rdf:resource="#testrequirement"/>
  <earl:testcase rdf:resource="#testcase"/>

  <earl:result rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#pass"/> 
</earl:Assertion>

earl:requirement --> required

A pointer to the Requirement that is being tested

earl:testcase --> optional

A pointer to the Test Case (according to my understanding of them explained at the beginning of the message) that is used. It should provide general information (id, creator, version, etc.) and process information (preconditions, postconditions, input data... And specially, all the atomic test we use).


> It seems to me that the evidence class may be an important 
> part of this puzzle. If the earl:TestRequirement class truly 
> references a requirement, the earl:Evidence could class be 
> used to reference the specific tests that were carried out in 
> order to make a conclusion about the result.
>
> However, earl:Evidence call will reference other 
> earl:Assertion classes. So at some point in the chain, we 
> must have a direct reference to the specific test from an 
> assertion. Besides, this may also be useful for other usages 
> of EARL that may not be built around the requirement - test hierarchy.
 

Why not use an earl:TestRequirement class to truly reference a Requirement, an earl:TestCase class to truly reference a Test Case and an earl:Evidence class to truly reference an Evidence (if anybody knows what an evidence is ;o)?


> This means we end up with the same problem we had in the 
> first place: either we define that earl:TestRequirement (or 
> whatever we call that class) to reference either a 
> requirement *or* a test; or we resurrect earl:TestCase and 
> define each of the classes to reference separate things.
> 
> Personally I favor the first option because it avoids 
> confusion between the two classes. There may be also other 
> options that have not been raised yet. I invite you to voice 
> your opinions on the problem and possible solutions.

Personally I think the first option doesn't avoid confusion at all, quite the contrary.

If we agree that Test Requirements and Test Cases are clearly different things, then IMO it's obvious we must keep them separate. I can't imagine how calling requirement to something which is a test case, or vice versa, can avoid confusion. Let's try with a comparison:

- We now that apples and lemons are different things (classes)
- We decide to call all them apples (in theory to avoid confusion between apples and lemons)
- I want to do an apple pie
- I say "give me apples"
- I receive apples and lemons but I didn't realise (How could I?)
- My apple pie tastes really strange
- Now I'm really confused and frustrated because I can't use the apples class to do an apple pie (Apparently the apples class doesn't contain only apples ¿¿¿???)

> Finally, CarlosI also mentioned the evaluation methodology as 
> another axis to this problem but we have not discussed how 
> this fits in. Maybe it will become clearer in subsequent discussions.


In fact I don't have a clear view of where Methodology and Evidence match in all this mess.

Is Methodology something similar to my understanding of Test Case? Is it something wider? Is there any overlap between them? 

What in the hell is an Evidence? ;oD , could TestCase be a subclass of Evidence?


> [1] <http://lists.w3.org/Archives/Public/public-wai-ert/2006Jan/0032>


Regards,

CI.

 
--------------------------------------

Carlos Iglesias

Web Accessibility Unit - CTIC Foundation
Science and Technology Park of Gijón
33203 - Gijón, Asturias, Spain 

phone: +34 984291212
fax: +34 984390612
email: carlos.iglesias@fundacionctic.org
URL: http://www.fundacionctic.org

Received on Monday, 30 January 2006 18:00:20 UTC