Re: some questions about version information in the test suite

On 18 Jun 2010, at 09:00 , Michael Kay wrote:
> ...
> From this experience (and despite anything I've said in the past)  
> I'm inclined to favour the XSLT model: each "test" (at some level of  
> granularity) defines the conditions under which it is to be run, and  
> a single expected result. If you want a variant of the test to be  
> run under different conditions with a different result, that's a  
> separate test.


Reading through this thread again, it occurs to me that the design
in http://www.w3.org/XML/2008/xsdl-exx/ancillary/xsts-schema.sketch.xml
amounts to saying that a "test" in this sense is given by one
'expected' element, not by an 'instanceTest' or 'schemaTest' element,
despite their names.

In the current sketch, there is one (input, conditions, output)
triple for each 'expected' element:

   - For schema tests, the schema being tested is given by the
     (schemaDocument children of the) enclosing schemaTest element.
   - For instance tests, the schema is given by the schemaTest
     child of the enclosing testGroup element, and the instance
     document by the (instanceDocument child of the) enclosing
     instanceTest.
   - The conditions are given by the 'version' attribute.
   - The prescribed result is given by the 'validity' attribute.

Various mechanisms can be devised to make it possible to
inherit conditions usefully, but they are just syntactic sugar.
And conversely, in the current syntax making it impossible to
inherit input information would, it seems to me, amount to
syntactic wormwood and gall.

It seems to me there are three areas that need further work and
clarification:

(1) We need to distinguish cleanly between conditions that
determine whether a test is applicable and should be run
(or the parameters with which it should be run) and the
conditions which determine different prescribed results when it
is run.

In the case of XSD, as far as I can tell every schema test is
applicable to all schema processors.  Every schema using a
1.1 construct serves as a test of a 1.0 processor's check on
the validity and conformance of its input schemas.

For instance tests, the most obvious condition of applicability
is whether, for the given configuration, the schema is judged
to conform or not.  There may be others, but this one seems to
be constant:  non-conforming schema, no meaningful instance
test.

(2) We need to decide how to specify both applicability
conditions and conditions on results; as I think about it,
I more and more lean toward the idea of defining (in the
schema, or informally) a set of keywords which can be
interpreted as true or false for a given configuration of
a given implementation, and using a white-space delimited
list of tokens to indicate either the conjunction or the
disjunction of those conditions.

Then we add a new token to the list when and if we need
it, documenting in human-readable form just what it means.
This will require human intervention in setting up a test
suite, but I don't see how that can be avoided.

(3) We need to decide just how and whether to inherit information
about applicability and/or conditions.


Michael

-- 
****************************************************************
* C. M. Sperberg-McQueen, Black Mesa Technologies LLC
* http://www.blackmesatech.com
* http://cmsmcq.com/mib
* http://balisage.net
****************************************************************

Received on Friday, 18 June 2010 22:03:36 UTC