RE: expected result -- too simple?

If your goal is "to see whether different processors construct the same
schema from the set of schema documents, or different schemas." then perhaps
the facility you're missing isn't the ability to make assertions about the
PSVI, it's rather the ability to make assertions about the schema
components; and rather than having an XML representation of the PSVI,
perhaps we need an XML representation of the schema components.

Either way, you seem to be asking for a substantial increase in the
complexity of the test framework, which is not necessarily a good thing, at
a time when we seem be stretched for resources; additional complexity in
some of the tests would seem preferable. Presumably you can automate the
construction of your 81 tests.

Michael Kay

> -----Original Message-----
> From: public-xml-schema-testsuite-request@w3.org 
> [mailto:public-xml-schema-testsuite-request@w3.org] On Behalf 
> Of C. M. Sperberg-McQueen
> Sent: 08 September 2007 17:40
> To: public-xml-schema-testsuite@w3.org
> Cc: C. M. Sperberg-McQueen
> Subject: expected result -- too simple?
> 
> 
> As an exercise, I'm trying to describe some tests I created 
> yesterday using the test suite catalog vocabulary.
> 
> I've run into what seems like a problem, either in my 
> conception of test construction or in the definition of the 
> vocabulary.
> 
> My tests relate to schema composition and are intended first 
> and foremost to illuminate questions of interoperability 
> among schema processors.  As currently formulated, they 
> specify a set of schema documents and then use the schema 
> thus constructed to validate a single test instance.  One 
> goal of the tests is to see whether different processors 
> construct the same schema from the set of schema documents, 
> or different schemas.
> So the test instance contains a number of elements which will 
> be valid or invalid, depending on what the processor has made 
> of the schema documents.  By seeing which elements are 
> accepted as valid and which elements are rejected as invalid, 
> it's possible to tell with a certain amount of confidence 
> which components are actually in the schema constructed by 
> the processor, and what their properties are.
> 
> Fuller details on the construction of the tests can be found 
> in section four of a working paper I'm drafting, at 
> http://www.w3.org/XML/Group/2007/09/schema_composition/model.xml
> (member-only link).
> 
> The problem is this:  for convenience I've constructed the 
> test as a single wrapper element containing several children
> -- eighty-one children, in the current form of the test.  If 
> we can clarify exactly what the spec requires in a given 
> case, the expected result will be a particular pattern of 
> [validity] properties on those eighty-one children.  But the 
> test suite catalog vocabulary does not allow such a 
> specification of what is expected:  it seems only to care 
> about the [validity] property (and not even the [validation 
> attempted] property?!) on the validation root (which is I 
> assume expected to be the document root element).
> 
> For purely practical reasons I am hesitant to take each of my 
> current tests and replace it with eighty-one tests.  (In some 
> cases, an analysis of the specific schema documents in the 
> test group could reduce the number of tests required to a 
> smaller number, but that would complicate the process of 
> constructing the tests and would be error prone.)
> 
> Why is 'expected' restricted to specifying the expected value 
> of a single property on a single element, instead of a larger 
> set of properties on all elements in the test instance?  Does 
> such a reduction really produce a sufficient reflection of 
> conformance?  I suppose it's harder to specify a fuller set 
> of expected results, and harder to check actual results for 
> agreement with expectations (given that we don't have a 
> standard XML representation or API for PSVI information).
> 
> In the long term, I wonder if we need to re-think the nature 
> of the expected test results.  In the short term, it would at 
> least be nice to be able to specify the expected validity of 
> each element in the test instance.
> 
> Is there a way around this problem, or should I resign myself 
> for now to having eighty-one times as many tests as I expected?
> 
> --C. M. Sperberg-McQueen
>   
>

Received on Monday, 10 September 2007 08:23:05 UTC