'Re: "RE: The dubious XML schema test collection"'

Hi,

on 9/2/2004 1:50 PM Michael Kay wrote:

> The test collection certainly has its faults, but I found it an invaluable
> resource when developing the schema processor in Saxon-SA. Yes, it would be
> nice if it were better, but I don't believe one should look a gift horse in
> the mouth. I certainly don't believe that it does more harm than good.

Yes, I feel that a test suite is invaluable indeed - I intended to 
badge. I'm thankfull for every peace of test to rely upon. But the suite 
is out there for 2-3 years now it seems; it's not marked at all to be 
incorrect in some parts. There is just a table of results showing the 
discrepancy of results produced by various XML Schema processors. We can 
add another column with our results to it and it would not gain a bit.

> I didn't use the test definition files or the reference results myself: I
> just tried to process all the schemas and validate the relevant documents
> against them, comparing the results of my processor with that of other
> processors. This is sometimes a bit hit-and-miss (you can reject an invalid
> schema for the wrong reason) but it exercises the processor reasonably

This defines a "XML schema with corresponding XML document collection" 
not a test-collection.

> thoroughly and experience in the field suggests that (with the help of some
> carefully planned supplementary testing) it got most of the bugs out.
> 
> Despite the very large numbers of tests, there are some areas where coverage
> is not good. For example, I think there are only two tests that redefine a
> schema with a change of namespace.
> 
> I'm not sure what you mean when you say some of the tests are "incorrect".

"Incorrect" should mean that the XML Schema documents and the 
corresponding XML documents, regarding their expected validation 
results, do not conform to the spec.

> Of course, there are many invalid schemas there, as there should be. Also,

Yes.

> some of them are invalid because they use obsolete syntax that was changed
> before the final Rec came out.

So they are broken.

> The three groups of tests complement each other quite well. The Sun suite is
> a small set of tests that's quick to run, but manages quite a high coverage
> of the spec. The NIST suite goes into exhaustive detail on testing the
> validation of simple types. The Microsoft tests are very large in number and
> manage a pretty broad coverage, though many of them are testing trivial
> error conditions like dangling references, and the coverage of the deeper
> semantic issues (like UPA) is much weaker.

Yes, the NIST tests seem to be very detailed - and correct.

> I would love to share some of the improvements I have made to the test suite
> but I simply don't have the time. I'm sure the same goes for other people
> including the original contributors.

And this I can completely understand. Maby there is need to create some 
room for people who have the time actually.

The conclusion for me:

I'll communicate "incorrect" tests to this mailing list when
   encountered.

Special offer:

If the test initiative needs the complete MS and SUN test definition 
files after 2 (3?) years, I can send a copy.

Proposals:

1 Mark the test-collection clearly as "work-in-progress" and incorrect
   in some parts.

2. It should not be communicated to try to conform to the MS tests
   naively (as long as it is not intended to create
   non-spec-conformant-but-MS-conformant schema processors)


Regards,

Kasimier

Received on Thursday, 2 September 2004 12:50:08 UTC