Re: early version of the XML Query Test Suite

Andrew Eisenberg wrote:
> 
> The XML Query Working Group and the XSL Working Group would like to 
> announce the availability of an early version of the XML Query Test 
> Suite (XQTS) at http://www.w3.org/XML/Query/test-suite/.

I welcome and appreciate this test suit.  However, I'm missing a 
rationale for the way it is laid out and how tests are written.  It 
seems an awfully inconvenient and hard-to-use format.  Each test is 
spread out over at least 3 files: the xquery, the expected result, and 
the catalog entry.  The latter is an entry in a huge xml file, and all 3 
are in different directories.  This makes it very inconvenient to add a 
testcase, and also very inconvenient to understand, read, or debug an 
individual testcase.

This is very much contrary to the principles of both agile programmng 
and good regression-testing practices.  For the latter one should add a 
new test case when fixing a bug.  In that case the meta-information, 
test program/query, and expected result *really* need to be in the same 
directory, and it's much easier to manage if they're all in a single file.

A much more "user-friendly" approach is to have the meta-information and 
the expected result in the same file as the query or at least allow that 
  as an option for short queries.  There are many ways to do that:
- Put the meta-information and expected result in comments.
- Put everything, including the query, in an xml file.  The objection is 
that neither the query nor result are necessarily XML.  Well, that is 
what we have CDATA for.  (The Qexo testsuite allows XML elements in the 
query or result.  The effective query is the serialization of the query 
node of the test case, but without escaping.   That works quite well.)
- Use a non-XML format, with delimiters, like Bumblebee.

Note one might still have multiple test cases in a single file; the 
point is that each test case is in one and only one file; not that each 
file contains one and only one test case.

Note also it is not clear that a catalog is needed or desirable.  It is 
easier to add a new testcase by adding a new file, without having to 
modify (patch) an existing file.  We automatically create and update a 
catalog by putting a new named file in a directory. Such an implicit 
catalog simplifies development and test management.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Received on Friday, 12 August 2005 06:47:45 UTC