expected result -- too simple?

As an exercise, I'm trying to describe some tests I created
yesterday using the test suite catalog vocabulary.

I've run into what seems like a problem, either in my
conception of test construction or in the definition of
the vocabulary.

My tests relate to schema composition and are intended first
and foremost to illuminate questions of interoperability
among schema processors.  As currently formulated, they specify
a set of schema documents and then use the schema thus constructed
to validate a single test instance.  One goal of the tests is
to see whether different processors construct the same schema
from the set of schema documents, or different schemas.
So the test instance contains a number of elements which
will be valid or invalid, depending on what the processor
has made of the schema documents.  By seeing which elements are
accepted as valid and which elements are rejected as invalid,
it's possible to tell with a certain amount of confidence which
components are actually in the schema constructed by the
processor, and what their properties are.

Fuller details on the construction of the tests can be found
in section four of a working paper I'm drafting, at
http://www.w3.org/XML/Group/2007/09/schema_composition/model.xml
(member-only link).

The problem is this:  for convenience I've constructed the
test as a single wrapper element containing several children
-- eighty-one children, in the current form of the test.  If
we can clarify exactly what the spec requires in a given case,
the expected result will be a particular pattern of [validity]
properties on those eighty-one children.  But the test suite
catalog vocabulary does not allow such a specification of
what is expected:  it seems only to care about the [validity]
property (and not even the [validation attempted] property?!)
on the validation root (which is I assume expected to be the
document root element).

For purely practical reasons I am hesitant to take each of my
current tests and replace it with eighty-one tests.  (In
some cases, an analysis of the specific schema documents in
the test group could reduce the number of tests required to
a smaller number, but that would complicate the process of
constructing the tests and would be error prone.)

Why is 'expected' restricted to specifying the expected value
of a single property on a single element, instead of a larger
set of properties on all elements in the test instance?  Does
such a reduction really produce a sufficient reflection of
conformance?  I suppose it's harder to specify a fuller set
of expected results, and harder to check actual results for
agreement with expectations (given that we don't have a
standard XML representation or API for PSVI information).

In the long term, I wonder if we need to re-think the nature
of the expected test results.  In the short term, it would
at least be nice to be able to specify the expected validity
of each element in the test instance.

Is there a way around this problem, or should I resign myself
for now to having eighty-one times as many tests as I expected?

--C. M. Sperberg-McQueen

Received on Saturday, 8 September 2007 16:39:56 UTC