Re: some questions about version information in the test suite

Let's look at how this is done in the XSLT and XQuery test suites and 
how well it works there.

In XSLT, the model is that each test defines the conditions under which 
it is to be run: for example "only run this test if you have a 1.1 
schema-aware processor that does static type checking", or similar. If 
the same test can be run under different conditions, producing different 
results, then you make a copy of the test with a different expected 
result and a different set of conditions. I think this method works 
quite well. There's potential for tests multiplying out of control, but 
in practice this doesn't happen because relatively few tests have 
complex dependencies. Multiple expected results for a test are not 
allowed, except that a negative test can define a list of acceptable 
error codes.

In XQuery the model has been until recently that each test defines 
multiple acceptable results, without any indication as to which result 
is appropriate under which conditions. This has led to some problems, 
for example where one of the possible results is an error, but the error 
case only applies to processors (for example) that don't implement the 
ancestor axis. This can easily lead to test results that appear OK, 
where actually the processor threw a quite unrelated error. There's a 
second mechanism in the XQuery test suite, which is that tests dependent 
on some optional features (say static typing) are grouped under a 
particular node in the hierarchy of tests; test drivers can hard-code a 
decision not to run the tests in a particular branch of the hierarchy. 
This has been supplemented by a third mechanism, which currently applies 
only to the XQuery language version, where the expected results are 
labelled with the associated language version. A fourth mechanism has 
been introduced in the XQuery Update tests, and might find its way into 
the main XQuery suite, which is more like the XSLT mechanism: tests can 
be labelled as being dependent on particular optional features in the 
spec, such as revalidation.

One advantage of the XSLT model is that test drivers can interpret the 
dependency information in two ways. If the test depends, say, on the 
processor being schema-aware, the test driver can either say "this 
processor is not schema-aware, so don't run the test", or it can say 
"this processor can be configured to be schema-aware, so set the 
relevant configuration switches before running the test". Each test is 
run either once, or not at all. With the XQuery model, it's much harder 
to write a driver that runs the same test under multiple different 
conditions, for example once with a 1.0 processor and once with a 1.1 
processor.

 From this experience (and despite anything I've said in the past) I'm 
inclined to favour the XSLT model: each "test" (at some level of 
granularity) defines the conditions under which it is to be run, and a 
single expected result. If you want a variant of the test to be run 
under different conditions with a different result, that's a separate test.

Michael Kay
Saxonica

> Comments on the proposals at
>
>   http://www.w3.org/XML/2008/xsdl-exx/ancillary/xsts-schema.sketch.xml
>
> are still welcome.  I think they address more or less successfully
> the problems I have found most troubling when I've tried to use
> the current schema, but I don't know whether they address any
> problems others have had.
>
> On the question at hand, the sketch at the URI just given says
> that
>
>   - The testSuite/@schemaVersion attribute is documentary
>     only, for human consumption, and has no expected effect
>     on machine processing.
>
>   - The @version attribute on the 'expected' element specifies
>     the versions for which the result indicated is the
>     prescribed result.
>
>   - The expected/@version attribute inherits its default value
>     from the enclosing test element, which inherits in turn
>     from testGroup, which defaults to the meaning 'all versions'.
>
>   - A default value may be overridden by a more specific value
>     elsewhere, so if a testGroup has version="1.0 1.1 1.2 2.0"
>     and a test within it has
>
> <expected validity="indeterminate" />
> <expected validity="invalid" version="1.2" />
> <expected validity="valid" version="1.0-1e"/>
>
>     the implication is that the prescribed 1.2 result is
>     'invalid' and the prescribed 1.0 First Edition result
>     is 'valid' and in all *other* versions (including
>     1.0 Second Edition) it's 'indeterminate'.
>
>     That seems in principle like a nice counterpart to the
>     most-specific rule pattern familiar from XSLT and from CSS,
>     but it may be too fancy and sophisticated and require too much
>     work, so I'm not sure it's a good idea.  It would be good to
>     have views of others involved in writing 1.1 tests or
>     running 1.1 tests.
>
>     In particular, I have not been able to think of any way
>     of handling differences between editions that does not
>     require either a little intelligence in setting up a
>     test harness to know about the relations among "1.0",
>     "1.0-1e", and "1.0-2e", or else a lot of very tedious
>     repetition of values in test cases where both editions
>     prescribe the same result.
>
> It seems to me that these changes produce a somewhat clearer
> and more usable account of the 'version' attributes than the
> current design.  But of course the person who writes the first
> sketch of the documentation always thinks it's clear :)
>

Received on Friday, 18 June 2010 15:01:01 UTC