Re: Testing the Conformance Testing tools

olivier Thereaux wrote:
> 
> Dear all,
> 
> some of you may remember an earlier mail thread [1] on this list, where 
> we discussed a list of test cases validome is using to compare their 
> tool to other validators. I thought at the time (and still do) that this 
> was an excellent list, but that I would like to be able to see 
> authoritative sources for what the expected result would be, making the 
> list more of a test suite, and less of a marketing tool.
> 
> [1] 
> http://lists.w3.org/Archives/Public/www-validator/2006Apr/thread.html#msg67
> 
> Having a test suite for conformance testing tools is extremely 
> important, for a whole slew of reasons:
>  - Specification are not always perfect and it is sometimes difficult 
> for conformance testing tools to interpret them.
>  - Often, conformance to a specification goes well beyond validity 
> against a grammar. Sometimes, there isn't even a grammar
>  - Some specifications don't have test suites. When they do, they often 
> are useful to test what is supposed to work. Conformance testers are in 
> the realm of dealing with what doesn't conform, what doesn't work. In 
> other words, most test suites are focusing on other classes of products 
> than testing tools, and testing tools need their own test suites.
> 
> In discussions with Alex of Validome, we talked about this topic, and 
> through these discussions, the idea to make a framework that would 
> manage test suites for markup language validators and conformance 
> checkers was born.
> 
> The basic idea is to create a repository of test cases that would make 
> it possible to create simple test cases: a document, an expected 
> conformance checking result, and references/argumentation for the 
> expected result. If input to the framework can be collectively 
> contributed to, even better. Of course, a repository of test isn't very 
> useful if it can't be fed to a system that can run the tests (e.g [2]) 
> and compare the output of one (or more) testing tool(s) to what is 
> expected.
> 
> [2] http://dev.w3.org/cvsweb/2002/css-validator/autotest/
> 
> 
> The Validome team, I think, is already starting work on this, and the 
> result will be open and open source. This would be a great opportunity 
> for cooperation, and I am sure that the participants in this list who 
> are also developing and maintaining such tools (Hello Christoph, Jirka, 
> Nick, Sam, Henri etc.) would be primary customers for it, and would be 
> interested in participating - either in the development of the test 
> framework, or in the building of test suites for testing language X or Y.
> 
> I have already invited the validome team to get cvs accounts here at 
> W3C, which I think would be a good place for such development, as a good 
> meeting ground, and for the insurance that resources hosted here will be 
> persistent and public. I would like to extend that invitation to others 
> interested in participating in this project. Please contact me if you are.

I'm interested.

- Sam Ruby

Received on Wednesday, 25 October 2006 11:46:26 UTC