RE: early version of the XML Query Test Suite

In my own test suites I've tended to use an approach where queries are
self-contained and there is no catalog; instead there are implicit
relationships between files based on naming conventions and ad-hoc control
files e.g. to specify whether source documents should be validated or not.
I've been using XQTS during its development and I have to say I've found the
catalog approach much more manageable than my previous ad-hoc system.

For my XSLT tests, I'm moving slowly to a middle ground: the catalog exists,
but is generated from data contained in the test files. This is easier to do
with XSLT, because stylesheets are XML and can therefore be queried and
transformed, and can contain XML metadata in a user-defined namespace. It's
hard to see how this could be made to work in XQuery except by the CDATA
approach, which to my mind would add more overhead than the free-standing
catalog.

The great advantage of having the catalog is that it forces test developers
to create metadata that conforms to some standard, which can be checked by
means of schema validation, and once you've done it it becomes easy to
perform operations on the test suite as a whole, such as finding all the
tests for a given error condition.

Michael Kay
http://www.saxonica.com/ 


> -----Original Message-----
> From: www-ql-request@w3.org [mailto:www-ql-request@w3.org] On 
> Behalf Of Per Bothner
> Sent: 12 August 2005 07:47
> To: Andrew Eisenberg
> Cc: www-ql@w3.org
> Subject: Re: early version of the XML Query Test Suite
> 
> 
> Andrew Eisenberg wrote:
> > 
> > The XML Query Working Group and the XSL Working Group would like to 
> > announce the availability of an early version of the XML Query Test 
> > Suite (XQTS) at http://www.w3.org/XML/Query/test-suite/.
> 
> I welcome and appreciate this test suit.  However, I'm missing a 
> rationale for the way it is laid out and how tests are written.  It 
> seems an awfully inconvenient and hard-to-use format.  Each test is 
> spread out over at least 3 files: the xquery, the expected 
> result, and 
> the catalog entry.  The latter is an entry in a huge xml 
> file, and all 3 
> are in different directories.  This makes it very 
> inconvenient to add a 
> testcase, and also very inconvenient to understand, read, or debug an 
> individual testcase.
> 
> This is very much contrary to the principles of both agile programmng 
> and good regression-testing practices.  For the latter one 
> should add a 
> new test case when fixing a bug.  In that case the meta-information, 
> test program/query, and expected result *really* need to be 
> in the same 
> directory, and it's much easier to manage if they're all in a 
> single file.
> 
> A much more "user-friendly" approach is to have the 
> meta-information and 
> the expected result in the same file as the query or at least 
> allow that 
>   as an option for short queries.  There are many ways to do that:
> - Put the meta-information and expected result in comments.
> - Put everything, including the query, in an xml file.  The 
> objection is 
> that neither the query nor result are necessarily XML.  Well, that is 
> what we have CDATA for.  (The Qexo testsuite allows XML 
> elements in the 
> query or result.  The effective query is the serialization of 
> the query 
> node of the test case, but without escaping.   That works quite well.)
> - Use a non-XML format, with delimiters, like Bumblebee.
> 
> Note one might still have multiple test cases in a single file; the 
> point is that each test case is in one and only one file; not 
> that each 
> file contains one and only one test case.
> 
> Note also it is not clear that a catalog is needed or 
> desirable.  It is 
> easier to add a new testcase by adding a new file, without having to 
> modify (patch) an existing file.  We automatically create and 
> update a 
> catalog by putting a new named file in a directory. Such an implicit 
> catalog simplifies development and test management.
> -- 
> 	--Per Bothner
> per@bothner.com   http://per.bothner.com/
> 
>

Received on Friday, 12 August 2005 09:42:18 UTC