Testing the Conformance Testing tools

Dear all,

some of you may remember an earlier mail thread [1] on this list,  
where we discussed a list of test cases validome is using to compare  
their tool to other validators. I thought at the time (and still do)  
that this was an excellent list, but that I would like to be able to  
see authoritative sources for what the expected result would be,  
making the list more of a test suite, and less of a marketing tool.

[1] http://lists.w3.org/Archives/Public/www-validator/2006Apr/ 
thread.html#msg67

Having a test suite for conformance testing tools is extremely  
important, for a whole slew of reasons:
  - Specification are not always perfect and it is sometimes  
difficult for conformance testing tools to interpret them.
  - Often, conformance to a specification goes well beyond validity  
against a grammar. Sometimes, there isn't even a grammar
  - Some specifications don't have test suites. When they do, they  
often are useful to test what is supposed to work. Conformance  
testers are in the realm of dealing with what doesn't conform, what  
doesn't work. In other words, most test suites are focusing on other  
classes of products than testing tools, and testing tools need their  
own test suites.

In discussions with Alex of Validome, we talked about this topic, and  
through these discussions, the idea to make a framework that would  
manage test suites for markup language validators and conformance  
checkers was born.

The basic idea is to create a repository of test cases that would  
make it possible to create simple test cases: a document, an expected  
conformance checking result, and references/argumentation for the  
expected result. If input to the framework can be collectively  
contributed to, even better. Of course, a repository of test isn't  
very useful if it can't be fed to a system that can run the tests  
(e.g [2]) and compare the output of one (or more) testing tool(s) to  
what is expected.

[2] http://dev.w3.org/cvsweb/2002/css-validator/autotest/


The Validome team, I think, is already starting work on this, and the  
result will be open and open source. This would be a great  
opportunity for cooperation, and I am sure that the participants in  
this list who are also developing and maintaining such tools (Hello  
Christoph, Jirka, Nick, Sam, Henri etc.) would be primary customers  
for it, and would be interested in participating - either in the  
development of the test framework, or in the building of test suites  
for testing language X or Y.

I have already invited the validome team to get cvs accounts here at  
W3C, which I think would be a good place for such development, as a  
good meeting ground, and for the insurance that resources hosted here  
will be persistent and public. I would like to extend that invitation  
to others interested in participating in this project. Please contact  
me if you are.


regards,
olivier
-- 
olivier Thereaux - W3C - http://www.w3.org/People/olivier/
W3C Open Source Software: http://www.w3.org/Status

Received on Wednesday, 25 October 2006 06:59:36 UTC