Test Suite catalog

Norm Walsh asked me to see if XOM could pass the xml:id test suite. The 
first thing I noticed is that there's no driver document to indicate how 
a test is passed. I suggest something like this modeled after the OASIS 
XSLT conformance test suite:

<?xml version="1.0" encoding="UTF-8"?>
<test-suite>
<test-catalog submitter="Sun Microsystems">
<creator>Norm Walsh</creator>
<date>2005-03-28</date>
   <test-case category="????" id="normalize_001">
   <file-path>tests</file-path>
   <creator>Norm Walsh</creator>
   <date>2005-03-28</date>
   <purpose>Normalize xml:id attributes</purpose>
   <spec-citation place="4" type="section" version="1.0" spec="xml:id"/>
   <scenario operation="standard">
     <input-file>001_normalize.xml</input-file>
     <id>te st</id>
   </scenario>
</test-case>
...
</test-catalog>

</test-suite>

Possibly we could simplify this some since xml:id is much simpler than 
XSLT and the test suite is much smaller.

The main difference from the OASIS format is the use of id elements to 
specify the expected output rather than output files. Each scenario 
would contain a list of zero or more id elements. The value of each such 
element is the value that should be reported by a conformant processor. 
For instance the above test case requires that the processor normalize 
the white space in the original attribute. Order matters here. 
Processors would be required to generate the exact values shown in the 
exact order.

This makes writing a test driver fairly straight-forward. Load the 
catalog file. Iterate through all the tests. For each input file, list 
all the ID-type attributes in the order they appear and compare to the 
id elements in the scenario element.

There are only eleven test cases currently, so it shouldn't be too hard 
to put this together. I'll try make a first pass on the problem this 
weekend, and then we can edit it or play with the markup as people think 
appropriate.

Here are some things I'm not sure of yet:

1. Should we assume people will install the files on their local file 
systems or should we just point to the test cases with absolute URLs to 
the W3C site?

2. Do we list only xml:id values or all ID-type attribute values?

3. The test-case element should have a features attribute to identify 
optional features required to pass a particular test case that not all 
implementations may support. For instance, some of the test suites 
require schema support or XML 1.1 support, which not all processors will 
supply. Reporters should list these tests as skipped or "not applicable" 
rather than failed. What is the list of such features?

4. Would the working group like to define a standard output format for 
submitting test results? I think the XInclude working group has a nice 
format we could borrow if so.

Comments? Questions? Concerns?


-- 
Elliotte Rusty Harold  elharo@metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim

Received on Saturday, 2 April 2005 14:03:53 UTC