Re: Test Cases and Comparisons Re: Minutes for Teleconference on 30 August, 2006

Hi Karl,

The current EARL model (which will be depicted in the next Editor's draft) looks as follows:

1. The first assumption is that a test case is an atomic test in the context of the test suite. The test case consists of the test description (aka test procedure) and the required files to execute the procedure. So, for each executed test case you need an assertion to express the outcome from the validator:

<earl:Assertion rdf:ID="assertionX">
  <earl:testcase rdf:resource="http://example.org/validator/css-tests/test-X" />
  ...
</earl:Assertion>


2. The second assumption is that there may be a grouping of test cases in the test suite to address requirements. For example, to test that the validator adequately addresses a specific CSS functionality, you may need to execute one or more test cases to determine. For each statement you want to make about such a requirement, you need a different "sort" of assertion:

<earl:Assertion rdf:ID="assertionY">
  <earl:requirement rdf:resource="http://example.org/css-spec/functionality-Y" />
  ...
</earl:Assertion>


With this you have statements about how the validator performed on the level of atomic test cases as well as a more higher-level view of how it complies with the target specification.

However, an important piece is missing. Namely a way to express how the test requirements and the test cases relate (i.e. the knowledge of "to make a statement about functionality Y, you need to execute tests A, B, and C"). We have decided that this is outside the scope of EARL and that it is to be defined in a test description language.

Note: We included the optional properties dc:title, dc:description, dc:hasPart, and dc:isPartOf in the earl:testable statement (super-)class. This provides a very rudementary method to describe tests until an appropriate test description language is available. Here is an example of using this:

<earl:testcase rdf:about="http://example.org/validator/css-tests/test-X">
  <dc:title xml:lang="en">Test Case X</dc:title>
  <dc:description>take file A and run it with parameters o,p, and q</dc:description>
  <dc:isPartOf rdf:resource="http://example.org/css-spec/functionality-Y" />
</earl:testcase>


We would be very happy to hear your feedback on this approach!


Regards,
  Shadi


Karl Dubost wrote:
> 
> Hi Shadi and ERT WG,
> 
> Le 30 août 06 à 23:56, Shadi Abou-Zahra a écrit :
>> Please find the minutes for the teleconference on 30 August, 2006:
>> <http://www.w3.org/2006/08/30-er-minutes>
> 
> [[[
> SAZ: Main issue is missing TCDL which limits the interopability to say 
> things are the same - we can get by without that and even in the worst 
> case you can still work with the general superclass of testablestatement 
> ... We hope people would be more consistent and a TCDL will arrive in 
> the future
> ]]] -- http://www.w3.org/2006/08/30-er-minutes
> Wed, 30 Aug 2006 14:51:38 GMT
> 
> In W3C [QA dev][1], we are developing a [Test Suite driver for CSS 
> Validator][2]. The intent is to test the validator itself.
> 
> The way it is working right now, there are two files
>     - a Test Case (CSS file or html file valid or not valid)
>     - a predefined output from the validator
>     - a description of the test
> 
> 1. The test case is sent to the validator for validation
> 2. The validator returns a message.
> 3. The message is compared to the predefined output.
> 4. From the comparison a pass or fail is issued
> 
> We were discussing with Olivier two days ago on how we would express 
> that with EARL, aka the link to the file, the supposed result, the 
> description and the Pass, Fail results.
> 
> I remember also that in SVG, they were using comparison images for their 
> Test Suite.
>     1. An image reference is given in PNG
>     2. a file in SVG is sent to a renderer.
>     3. A bitmap of this rendering is generated,
>     4. the bitmap and the reference image are compared.
> 
> 
> I see the last Editor's Draft is 18th of July[3].
> Could you give an example on how we would express this kind of pattern 
> in EARL, if possible?
> 
> 
> 
> [1]: http://www.w3.org/QA/Tools/qa-dev/
> [2]: http://dev.w3.org/cvsweb/2002/css-validator/autotest/
> [3]: http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060718
> 
> 
> --Karl Dubost - http://www.w3.org/People/karl/
> W3C Conformance Manager, QA Activity Lead
>   QA Weblog - http://www.w3.org/QA/
>      *** Be Strict To Be Cool ***
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
Shadi Abou-Zahra     Web Accessibility Specialist for Europe | 
Chair & Staff Contact for the Evaluation and Repair Tools WG | 
World Wide Web Consortium (W3C)           http://www.w3.org/ | 
Web Accessibility Initiative (WAI),   http://www.w3.org/WAI/ | 
WAI-TIES Project,                http://www.w3.org/WAI/TIES/ | 
Evaluation and Repair Tools WG,    http://www.w3.org/WAI/ER/ | 
2004, Route des Lucioles - 06560,  Sophia-Antipolis - France | 
Voice: +33(0)4 92 38 50 64          Fax: +33(0)4 92 38 78 22 |

Received on Thursday, 31 August 2006 16:06:16 UTC