Re: some questions about version information in the test suite

On 16 Jun 2010, at 10:17 , Henry S. Thompson wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> C. M. Sperberg-McQueen writes:
>
>> The test suite schema seems to have several attributes for dealing
>> with different versions of the schema language:
>>
>>  - the schemaVersion attribute on ts:testSuite
>>  - the version and edition attributes on schemaTest and instanceTest
>>  - the version and edition attributes on expected
>>
>> Q1 What do they mean, and what is the relation among them?
>>
>> The documentation in the test suite schema describes the
>> version attribute on schemaTest and instanceTest as indicating
>> that a particular test "applies to" particular versions
>> only.  I think I see what it means for an instance test to
>> be applicable only to a particular version of XSD -- if the
>> relevant schema is not legal in 1.0, the instance test is
>> meaningless for a 1.0 processor.   But what does it mean for
>> a schema test to be applicable only to certain versions of
>> the language?
>
> It means that the expected result is only correct per that/those
> stated version/edition(s).

I don't think I follow.  If a set of test suite catalog documents
has schemaVersion="x" on the testSuite element, version="y"
on an instanceTest element, and version="z" on the 'expected'
element within that instance test, where x, y, and z are
NMTOKENS expressions, does it mean

   (a) The test is applicable to the union of the sets of versions
       denoted by x, y, and z?
   (b) The test is applicable only to intersection of those sets
       of versions?

The schema document says that

   Test groups which only apply to certain versions of
   XML Schema list those versions in the version attribute.
   An absent version implies that the test group applies
   to all versions.

which suggests that if y="1.0" and z="1.1" then the test element
and the expected element are simply contradicting each other,
which would be incompatible with both interpretation (a)
and interpretation (b). So

   (c) The catalog document is contradicting itself.

It's also a little puzzling that the documentation quoted
seems to expect the attribute to appear on the testGroup
element, though it is given as documentation for the schemaTest
and instanceTest elements.

Speaking for myself, I find it difficult to write test case
metadata using a vocabulary whose semantics I don't understand.
(And judging by the rate at which other WG members have been
contributing 1.1 test cases, I can't help wondering whether
others are also finding it difficult.)  I have not succeeded
in writing any 1.1 test cases using the current design.

Hence my suggestion several days ago that we modify the
definition of the vocabulary to clarify the semantics and to
make the metadata structures easier to use for 1.1 tests.

Comments on the proposals at

   http://www.w3.org/XML/2008/xsdl-exx/ancillary/xsts-schema.sketch.xml

are still welcome.  I think they address more or less successfully
the problems I have found most troubling when I've tried to use
the current schema, but I don't know whether they address any
problems others have had.

On the question at hand, the sketch at the URI just given says
that

   - The testSuite/@schemaVersion attribute is documentary
     only, for human consumption, and has no expected effect
     on machine processing.

   - The @version attribute on the 'expected' element specifies
     the versions for which the result indicated is the
     prescribed result.

   - The expected/@version attribute inherits its default value
     from the enclosing test element, which inherits in turn
     from testGroup, which defaults to the meaning 'all versions'.

   - A default value may be overridden by a more specific value
     elsewhere, so if a testGroup has version="1.0 1.1 1.2 2.0"
     and a test within it has

       <expected validity="indeterminate" />
       <expected validity="invalid" version="1.2" />
       <expected validity="valid" version="1.0-1e"/>

     the implication is that the prescribed 1.2 result is
     'invalid' and the prescribed 1.0 First Edition result
     is 'valid' and in all *other* versions (including
     1.0 Second Edition) it's 'indeterminate'.

     That seems in principle like a nice counterpart to the
     most-specific rule pattern familiar from XSLT and from CSS,
     but it may be too fancy and sophisticated and require too much
     work, so I'm not sure it's a good idea.  It would be good to
     have views of others involved in writing 1.1 tests or
     running 1.1 tests.

     In particular, I have not been able to think of any way
     of handling differences between editions that does not
     require either a little intelligence in setting up a
     test harness to know about the relations among "1.0",
     "1.0-1e", and "1.0-2e", or else a lot of very tedious
     repetition of values in test cases where both editions
     prescribe the same result.

It seems to me that these changes produce a somewhat clearer
and more usable account of the 'version' attributes than the
current design.  But of course the person who writes the first
sketch of the documentation always thinks it's clear :)

-- 
****************************************************************
* C. M. Sperberg-McQueen, Black Mesa Technologies LLC
* http://www.blackmesatech.com
* http://cmsmcq.com/mib
* http://balisage.net
****************************************************************

Received on Friday, 18 June 2010 14:29:02 UTC