Testable assertion tagging for W3C specifications

A colleague and I have been talking about how to do more precise testing of
W3C specifications, and how the specification markup might help.

GOALS:
   Allow an external document (test case, erratum, email, etc.) to point
   directly at a "testable" normative sentence in a Recommendation.
   Encourage document editors to view some of the sentences as "test
   assertions" and to write them in a style that conveys precisely what
   they declare.
   Explore possibilities for machine processing of testable sentences in
   the future.
   Link error assertions to error catalogues (see the work that Mike Kay is
   doing with the XSLT document: (
   http://www.w3.org/TR/xslt20/#error-summary)).
   Provide a tagging scheme for testing of grammatical statements, such as
   the ad-hoc one employed in the XPath/XQuery specifications.
   Possibly provide markup also for discretionary behavior.

So our proposal is to add a tagging structure to
http://www.w3.org/XML/1998/06/xmlspec-report.htm,
http://www.w3.org/XML/1998/06/xmlspec.dtd, that allows editors to achieve
the above goals.

Test cases will nearly always have to cite more than one testable sentence
and/or production, unless the Rec is issued with test assertions in a
separate appendix. We should experiment with enhanced tagging to see how it
influences sentence structure. Some complex sentences with multiple "or"
parts crossing each other may get restructured just to make citing them
more precise. Consider this sentence from part 16.4 of XSLT 1.0: "Thus, it
is an error to disable output escaping for an xsl:value-of or xsl:text
element that is used to generate the string-value of a comment, processing
instruction or attribute node; it is also an error to convert a result tree
fragment to a number or a string if the result tree fragment contains a
text node for which escaping was disabled." That one sentence has 8-10
testable assertions.

So far, we are not proposing concrete details.  First we wanted to see what
people thought of the idea, if anyone has experimented with something like
this so far, and whether or not this would be worth a concrete proposal.

-scott

Received on Monday, 6 May 2002 13:38:24 UTC