Re: some questions about version information in the test suite

On 16 Jun 2010, at 09:53 , Henry S. Thompson wrote:

 > C. M. Sperberg-McQueen writes:

 >> (2) eliminate the implDe attribute, and (3) change the enumerated
 >> values of expected/@validity from 'valid, invalid, notKnown' to
 >> 'valid, invalid, notKnown, impl-defined, impl-dependent,
 >> unspecified'

 > I will respond at more length to the original message, but wrt this
 > proposed change I would point out that the WG considered a similar
 > change six months ago and decided against it,

Then if we were operating under the same rules for the test suite as
we use for the spec, I would say: I'm reopening the issue because I
was not present when that decision was made and the line of reasoning
that leads me to my conclusion was, I believe, not considered by the
WG.  We aren't in fact operating under those rules, so we are not
bound by that earlier decision, so I'll say no such thing, only that I
think the test suite schema is broken and we need to fix it.

 > because the current values of expected/@validity correspond to the
 > three allowed values of the [validity] PSVI property, and are
 > intended to be checked against it.

The conclusion doesn't seem to me to follow from the premise.

That the three values 'valid', 'notKnown', and 'invalid' are the legal
values of the [validity] property seems like a good reason to make
them the legal values of the testResult element's validity attribute.

They are not a complete enumeration of what the XSD spec prescribes as
a result for validation, and they are not suitable as the sole
possible values for expected result.  They are not in fact ideal as an
exhaustive set of values even for testResult/@validity, given that in
some cases the test result for an instance test ought ideally to be
able to disinguish 'error: non-conforming schema' from 'instance is
not valid'.  Also, the terms 'valid', 'invalid', and 'notKnown' are
not defined as properties of schemas, which have no [validity]
property, so this triad is not well matched to the task of enumerating
the results of schema tests, either.  In the interests of backwards
compability I don't propose to change the practice of using 'valid'
and 'invalid' to mean 'conforming' and 'non-conforming' in direct
contradiction to the definitions in the XSD spec, but the mismatch
does suggest that the argument from the [validity] property is
unsound.

One problem with the current design is that it suggests that there is
or could be some meaningful difference between a test with an expected
result of 'valid' and an implDe value of 'true', and a test with an
expected result of 'invalid' and an implDe value of 'true', but
nothing explains what that meaningful difference is.  (Informally,
I've heard one suggestion which amounts to "when @implDe = true, then
expected/@validity identifies not the prescribed result of the test,
but records the value first proposed by the creator of the test case",
but nothing in the documentation supports that interpretation, it does
not work for cases where the creator of the test case marks it as
having implementation-defined results, and there are in any case other
better ways to record change history.)

The current design also places the implDe attribute on the test case
insted of on the expected result, thus suggesting wrongly that having
an implementation-defined, implementation-dependent, or otherwise
indeterminate result is a property of the test case, instead of being
a property of the test case evaluated using a particular version of
XSD (and with particular starting conditions for the validation).  We
are going to have to get rid of @implDe on the test element in any
case, to make the schema make sense.  We might as well get the
modeling clean while we're at it.

If you mean to suggest that the logic of interpreting test results is
simpler

With an implDe attribute, the logic (assuming you don't care what your
processor does for implementation-defined, implementation-dependent,
or indeterminate tests) is:

   Let A be the value of [validity] on the validation root.
   Let E be the value of expected/@validity.
   Let I be the value of @implDe on the test element.
   If A = E or I = true then pass; else fail.

With the larger set of enumerated values for expected/@validity, the
logic is:

   Let A be the value of [validity] on the validation root.
   Let E be the value of expected/@validity.
   If A = E or E in {implementation-defined, implementation-dependent,
     indeterminate} then pass; else fail.

In fact, any serious implementor is likely to care about what the
processor does for tests with implementation-defined results (you want
to make sure the code is doing what your documentation says it's going
to do, and doing the same thing each time), so in practice the logic
is going to be more complicated in any case.

 > We chose instead to continue interpreting that attribute as
 > specifying what the _author_ of the test expected, and then using a
 > separate attribute (doesn't have to be a boolean @implDe -- if the
 > range of cases you want to cover (impl-defined, impl-dependent,
 > unspecified) can really be adequately distinguished, then make them
 > the values of a defaults-to-vanilla attribute called @status or some
 > such).

Ah.  That might be the origin of the informal account I have heard.
It's not documented, which suggests that the change was never
completely implemented, and it makes no sense to me at all. Does the
author of the test have a special status in our test suite?  And
what should the author of the test do if the author does not expect
any of the values 'valid', 'invalid', or 'notKnown', precisely
because the author of the test is aware that the result is
implementation-defined or implementation-dependent?  The current
design as you describe it, like the original design, makes it
impossible for the test suite ever to incorporate without semantic
error the tests of schema composition described in

   http://cmsmcq.com/2007/schema_composition/model.xml

The tests can be included in the schema test suite without expected
results, and if we add expected results for the tests on which
the spec is indeterminate, the rule you say the WG adopted will
license the inference that the author of the tests expected the
result to be 'valid', or 'invalid', or 'notKnown', which is not
true.

I submit that the design given at

http://www.w3.org/XML/2008/xsdl-exx/ancillary/xsts-schema.sketch.xml

is simpler and cleaner, and offers the WG a chance to correct the
mistake made six months ago.


-- 
****************************************************************
* C. M. Sperberg-McQueen, Black Mesa Technologies LLC
* http://www.blackmesatech.com
* http://cmsmcq.com/mib
* http://balisage.net
****************************************************************

Received on Wednesday, 16 June 2010 18:02:18 UTC