some questions about version information in the test suite

The test suite schema seems to have several attributes for dealing
with different versions of the schema language:

   - the schemaVersion attribute on ts:testSuite
   - the version and edition attributes on schemaTest and instanceTest
   - the version and edition attributes on expected

Q1 What do they mean, and what is the relation among them?

The documentation in the test suite schema describes the
version attribute on schemaTest and instanceTest as indicating
that a particular test "applies to" particular versions
only.  I think I see what it means for an instance test to
be applicable only to a particular version of XSD -- if the
relevant schema is not legal in 1.0, the instance test is
meaningless for a 1.0 processor.   But what does it mean for
a schema test to be applicable only to certain versions of
the language?

When I started working on test cases for 1.1, I anticipated
being able to say things like "this schema has indeterminate
conformance in 1.0 [because the spec is a little vague on some
relevant point] and invalid in 1.1 [because the point was
clarified]" and "this schema is not legal in 1.0, but is
legal in 1.1", by writing, for example:

   <schemaTest name="snneg">
    <schemaDocument xlink:href="foo.xsd"/>
    <expected version="1.0" validity="invalid"/>
    <expected version="1.1" validity="valid"/>
   </schemaTest>

But the test suite schema appears to allow only one occurrence
of 'expected'.

Q2 Is this an oversight, or is it intended that it not be
possible to describe a given test as producing different
results in different versions of the language?

In the case of instance tests, one can, at some cost in
apparently unmotivated tedium, write

   <instanceTest name="foo">
    <instanceDocument xlink:href="foo.xml"/>
    <expected version="1.0" validity="implDe"/>
   </instanceTest>
   <instanceTest name="foo">
    <instanceDocument xlink:href="foo.xml"/>
    <expected version="1.1" validity="invalid"/>
   </instanceTest>

which is less clear and makes it harder to search the catalogs
for test cases exercising 1.0/1.1 changes.  But in the case
of schema tests, since testGroup allows at most one schemaTest
it would be necessary not only to duplicate the schemaTest element
but also to duplicate the enclosing testGroup element.

The test suite schema also appears not to allow the test creator
to say that the result of a given test is implementation-defined
in one version of the language but required to be valid, or
invalid, in another.

Q3 Is this an oversight or is it intended that it not be
possible to describe a given test as illustrating a construct
whose behavior is unspecified in 1.0 but specified in 1.1?

Q4 Can these problems be fixed?

I believe the simplest and best fix would be to (1) allow
'expected' to repeat, (2) eliminate the implDe attribute,
and (3) change the enumerated values of expected/@validity
from 'valid, invalid, notKnown' to 'valid, invalid, notKnown,
impl-defined, impl-dependent, unspecified' (where 'unspecified'
covers (a) cases where the version of the spec in question
is underspecified or vague, (b) where it is overspecified and
contradicts itself, and (c) where the WG cannot agree on what
the text of that version of the spec means -- in principle it
would be useful to disentangle these, but that might lead to
interminable wrangling over how to classify individual
cases).

An alternative which seems less good to me, because less clear,
would be to (1) allow 'expected' to repeat, and (2) move
the 'implDe' attribute from the test case to the 'expected'
element.

In either case, the documentation should make clear that it
is meaningless for an expected/@version attribute to include
a version identifier for a version excluded (i.e. not mentioned)
by the version attribute on the enclosing test element.

And if implDe is retained, it would be helpful to new students
of the vocabulary to make explicit that if @implDe is true,
the @validity attribute on the corresponding 'expected' element
has no defined meaning.

Q5 What form are the NMTOKEN values of the version and edition
attributes expected to take?

Q6 Why do the indications of version and edition used here not
refer to or exploit the version and edition identifiers defined
in the XSD 1.1 spec?

-- 
****************************************************************
* C. M. Sperberg-McQueen, Black Mesa Technologies LLC
* http://www.blackmesatech.com
* http://cmsmcq.com/mib
* http://balisage.net
****************************************************************

Received on Sunday, 13 June 2010 16:27:41 UTC