draft schema for test reports

Test suite fans,

It has taken a while, but I have now expanded the schema for test
catalogs to cover test reports as well.

There is of course no requirement that any implementor run any of the
test suites we develop, and no requirement that if they do, they use any
particular format for reporting their results.  But if we want to
compile the kind of table of test results many groups produce, showing
multiple implementations for different features of our spec, it will be
convenient if there is a common XML format for reporting test results.

The basic design is simple: for each container or test-case element in
the test catalog there is a corresponding element in the test report:

  test-catalog ~ test-report
  test-set, test-set-ref ~ test-set-results
  grammar-test ~ grammar-result
  test-case ~ test-result

The grammar-test and test-result elements carry a 'result' attribute
whose possible values are pass, fail, not-run, disputed, and other.  The
grammar-test and test-result elements may be empty, but they can also
contain metadata (description elements, pointers to external
documentation, and 'app-info' containing arbitrary additional
information) and details of the test.

The minimal form of test report should be straightforward to produce
from a JUnit test report, and either form of test report should be easy
to translate into JUnit if that is desired.  (But that's a 'should' not
'has been shown'.)

The schema can be found in the file lib/test-catalog.xml in my
ixml-tests repository [1], and a hand-created sample is in
doc/results-sketch.xml and doc/results-sketch.html [2, 3].

[1] https://github.com/cmsmcq/ixml-tests/blob/main/lib/test-catalog.rnc
[2] https://github.com/cmsmcq/ixml-tests/blob/main/doc/results-sketch.xml
[3] https://github.com/cmsmcq/ixml-tests/blob/main/doc/results-sketch.html

Comments and suggestions for improvement are welcome.

-- 
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com

Received on Tuesday, 8 February 2022 00:30:56 UTC