- From: Sam Ruby <rubys@intertwingly.net>
- Date: Wed, 25 Oct 2006 07:46:04 -0400
- To: olivier Thereaux <ot@w3.org>
- CC: www-validator Community <www-validator@w3.org>, Alex Leporda <al@validome.org>
olivier Thereaux wrote: > > Dear all, > > some of you may remember an earlier mail thread [1] on this list, where > we discussed a list of test cases validome is using to compare their > tool to other validators. I thought at the time (and still do) that this > was an excellent list, but that I would like to be able to see > authoritative sources for what the expected result would be, making the > list more of a test suite, and less of a marketing tool. > > [1] > http://lists.w3.org/Archives/Public/www-validator/2006Apr/thread.html#msg67 > > Having a test suite for conformance testing tools is extremely > important, for a whole slew of reasons: > - Specification are not always perfect and it is sometimes difficult > for conformance testing tools to interpret them. > - Often, conformance to a specification goes well beyond validity > against a grammar. Sometimes, there isn't even a grammar > - Some specifications don't have test suites. When they do, they often > are useful to test what is supposed to work. Conformance testers are in > the realm of dealing with what doesn't conform, what doesn't work. In > other words, most test suites are focusing on other classes of products > than testing tools, and testing tools need their own test suites. > > In discussions with Alex of Validome, we talked about this topic, and > through these discussions, the idea to make a framework that would > manage test suites for markup language validators and conformance > checkers was born. > > The basic idea is to create a repository of test cases that would make > it possible to create simple test cases: a document, an expected > conformance checking result, and references/argumentation for the > expected result. If input to the framework can be collectively > contributed to, even better. Of course, a repository of test isn't very > useful if it can't be fed to a system that can run the tests (e.g [2]) > and compare the output of one (or more) testing tool(s) to what is > expected. > > [2] http://dev.w3.org/cvsweb/2002/css-validator/autotest/ > > > The Validome team, I think, is already starting work on this, and the > result will be open and open source. This would be a great opportunity > for cooperation, and I am sure that the participants in this list who > are also developing and maintaining such tools (Hello Christoph, Jirka, > Nick, Sam, Henri etc.) would be primary customers for it, and would be > interested in participating - either in the development of the test > framework, or in the building of test suites for testing language X or Y. > > I have already invited the validome team to get cvs accounts here at > W3C, which I think would be a good place for such development, as a good > meeting ground, and for the insurance that resources hosted here will be > persistent and public. I would like to extend that invitation to others > interested in participating in this project. Please contact me if you are. I'm interested. - Sam Ruby
Received on Wednesday, 25 October 2006 11:46:26 UTC