W3C home > Mailing lists > Public > www-validator@w3.org > October 2006

Re: Testing the Conformance Testing tools

From: Sam Ruby <rubys@intertwingly.net>
Date: Wed, 25 Oct 2006 07:46:04 -0400
Message-ID: <453F4E7C.3060807@intertwingly.net>
To: olivier Thereaux <ot@w3.org>
CC: www-validator Community <www-validator@w3.org>, Alex Leporda <al@validome.org>

olivier Thereaux wrote:
> Dear all,
> some of you may remember an earlier mail thread [1] on this list, where 
> we discussed a list of test cases validome is using to compare their 
> tool to other validators. I thought at the time (and still do) that this 
> was an excellent list, but that I would like to be able to see 
> authoritative sources for what the expected result would be, making the 
> list more of a test suite, and less of a marketing tool.
> [1] 
> http://lists.w3.org/Archives/Public/www-validator/2006Apr/thread.html#msg67
> Having a test suite for conformance testing tools is extremely 
> important, for a whole slew of reasons:
>  - Specification are not always perfect and it is sometimes difficult 
> for conformance testing tools to interpret them.
>  - Often, conformance to a specification goes well beyond validity 
> against a grammar. Sometimes, there isn't even a grammar
>  - Some specifications don't have test suites. When they do, they often 
> are useful to test what is supposed to work. Conformance testers are in 
> the realm of dealing with what doesn't conform, what doesn't work. In 
> other words, most test suites are focusing on other classes of products 
> than testing tools, and testing tools need their own test suites.
> In discussions with Alex of Validome, we talked about this topic, and 
> through these discussions, the idea to make a framework that would 
> manage test suites for markup language validators and conformance 
> checkers was born.
> The basic idea is to create a repository of test cases that would make 
> it possible to create simple test cases: a document, an expected 
> conformance checking result, and references/argumentation for the 
> expected result. If input to the framework can be collectively 
> contributed to, even better. Of course, a repository of test isn't very 
> useful if it can't be fed to a system that can run the tests (e.g [2]) 
> and compare the output of one (or more) testing tool(s) to what is 
> expected.
> [2] http://dev.w3.org/cvsweb/2002/css-validator/autotest/
> The Validome team, I think, is already starting work on this, and the 
> result will be open and open source. This would be a great opportunity 
> for cooperation, and I am sure that the participants in this list who 
> are also developing and maintaining such tools (Hello Christoph, Jirka, 
> Nick, Sam, Henri etc.) would be primary customers for it, and would be 
> interested in participating - either in the development of the test 
> framework, or in the building of test suites for testing language X or Y.
> I have already invited the validome team to get cvs accounts here at 
> W3C, which I think would be a good place for such development, as a good 
> meeting ground, and for the insurance that resources hosted here will be 
> persistent and public. I would like to extend that invitation to others 
> interested in participating in this project. Please contact me if you are.

I'm interested.

- Sam Ruby
Received on Wednesday, 25 October 2006 11:46:26 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 1 March 2016 14:17:50 UTC