Re: some questions about version information in the test suite

On 18 Jun 2010, at 09:00 , Michael Kay wrote:

>
> Let's look at how this is done in the XSLT and XQuery test suites  
> and how well it works there.
>
> In XSLT, the model is that each test defines the conditions under  
> which it is to be run: for example "only run this test if you have a  
> 1.1 schema-aware processor that does static type checking", or  
> similar. If the same test can be run under different conditions,  
> producing different results, then you make a copy of the test with a  
> different expected result and a different set of conditions. I think  
> this method works quite well. There's potential for tests  
> multiplying out of control, but in practice this doesn't happen  
> because relatively few tests have complex dependencies. Multiple  
> expected results for a test are not allowed, except that a negative  
> test can define a list of acceptable error codes.

I agree entirely that we should adopt some mechanism similar to
those of XSLT and XQuery for dealing with optional features and
implementation-defined behaviors.  I think Mary Holstege may
have something to say on that topic when she reports on her
action.

Requiring that tests be duplicated in whole seems to me to make
it unnecessarily difficult to track and manage tests that focus
on differences among versions or differences among processors:
if we have multiple 'expected' elements with different conditions,
it is easy to find tests which exercise 1.0/1.1 differences:
search for tests with two 'expected' children, one with 1.0 and
one with 1.1 in the version attribute.  Searching a test suite
of a few tens of thousands of tests for another test which
happens to have the same schemaDocuments and the same instanceDocument
is much more awkward.

Since our requirement for getting to PR is to show that the
changes between 1.0 and 1.1 are implementable and have been
tested to show they have been successfully implemented, the
class of tests for which results differ on the same input
is crucially important to me.  Making those tests into pairs
or larger sets of tests is a large step toward making it harder
to get to PR.  Is there some compensating virtues that I am
missing?

-- 
****************************************************************
* C. M. Sperberg-McQueen, Black Mesa Technologies LLC
* http://www.blackmesatech.com
* http://cmsmcq.com/mib
* http://balisage.net
****************************************************************

Received on Friday, 18 June 2010 15:16:55 UTC