Sorry for jumping in on the thread at so late a stage. Being very interested in the topic and also being the principal author of the relevant parts of the Specification Guidelines document (which I co-edit), let me first say that I look forward to the results this discussion will lead to. Please include me in any relevant messaging, even off these lists. A word of warning; this is a rather long email, as I try to reply to and discuss as much as possible (of course, contrary to any implicit rule out there). General outline (in reply to Scott's original posting, parts of which are pasted, my comments prefixed by [dd]): --- GOALS: Allow an external document (test case, erratum, email, etc.) to point directly at a "testable" normative sentence in a Recommendation. [dd] This would clearly simplify the task of, if we look at tests, knowing which part in particualr is being tested, but requires structure and issues tracking. This in turn implies that it may need to be an intra-W3C "standard". Encourage document editors to view some of the sentences as "test assertions" and to write them in a style that conveys precisely what they declare. [dd] Any disambiguation is welcome. However, there are issues with, as I see it: 1. Forcing specification authors to write in a particular markup (which could be worth it, given the complexity they need to bea ble to master to begin with to author a spec) 2. Different WG's will certainly have their own views on how the xmlspe, for example, should be extended. Explore possibilities for machine processing of testable sentences in the future. [dd] This is already being done in an embryo-stage way, more on how we generate the DOM Test Suite markup language below. We have had positive responses from this approach, as it treats the specification (the XML version, actually) more normatively than just serving to produce the human readable HTML output. Link error assertions to error catalogues (see the work that Mike Kay is doing with the XSLT document: ( 1A1A,1A1A,FFFFhttp://www.w3.org/TR/xslt20/#error-summary)). Provide a tagging scheme for testing of grammatical statements, such as the ad-hoc one employed in the XPath/XQuery specifications. Possibly provide markup also for discretionary behavior. [dd] In general, I think the cost/benefit analysis woul end up with the follwing (The QA activity, of course, tries to emphasize the benefits, and for this and other reasons it is important that people out there raise their voice at an early stage) Costs 1. "another forced upon us way to write", as has already been discussed in this thread. People view the added cost as counter productive, given the amount of work they already do. My personal argument would be to look for the added benefits, which include: a. Easier machine processing of the specification, which allows for easier tracking (of tests, errata, issues, and so forth) b. Easier generation of different types of output (HTML for human reading, HTML+SVG for graphical representation, XML for implementations specific calls to the spec to do testing on the fly, and so forth) c. Added degree of normativity to (what I consider to be) the important portion of the specification: the XML document. This of course leads to digression, but is what taking the idea of enhancing the markup leads to eventually 2. Complicates authoring beyond need, even having adopted the extended markup. It is tedious to have to use intricate markup to write simple things. a. I don't think we'd ever want to use complicated markup fo write simple things, just add to what we do already. In the long run, it would certainly be beneficial to be able to let your implementation check whether it passes a test and report back to you instead of having to read the spec, interpret it, write the (proprietary, since there is no agreed on syntax) test, run it, interpret the result, re-reading the spec (and re-interpreting it, as we've all seen many times). The added benefit is that using intricate markup to write assertions allows for easier interpretation of the spec; in particular it greatly simplifies testing interdepencies between parts of specifications but also interdependent specifications (DOM/HTML, for example) 3. Not all specification aim at specifying the same thing, therefore extending a particular markup (which is not used by everyone, in any case) may not meet the aim. Benefits: 1. In general, I think the cost is short term, once adopted it will begin to show its advantages, making it more acceptable to new WG's and newer version of existing technologies. 2. Having an agreed on markup allows for separating between the assertions (where assertion is to be understood as David Marston explained earlier), the representation of those assertions (which we now view as the specification) and finally mechanisms for asserting tests, running them, reporting and so forth. In general, the aim to write implementable specifications and making it easier to implement them is furthered. 3. Machine calls can be used to ascertain compliance with particular specifications, thus lessening the discussion on details in specifications done by the same people who wrote it to begin with. --- DOM Framework: One example of many is what we did in the DOM TS framework (http://www.w3.org/DOM/Test). Here we use XSLT transforms to generate the markup language directly from the XML documents that serve to produce the HTML output. The TS (in XML) markup mimics code, includes pointers to what part of the specification the test tests, and so forth. XSLT is used, again, to produce the two official bindings (ECMA and Java), but other ones have been produced too (one C++ binding). This means that one test case description generates many test instances (one per binding). Furthermore, it is straightforward to group tests to form specific test suites (DOM allows for different setups, so to speak, for example entity handling) so that one can ensure that the implementation is being tested on its own merits. Finally, test reporting is greatly enhanced. The benefits we've seen is that the final deliverable (one TS per level of the DOM) is made easier, _once_ people come over the threshold of using a particular syntax (which speaks for the fact that there is resistance to changing tools, so to speak). However, once people saw that the new framework was worth supporting, more an more actors (re)wrote tests in order to comply with the framework. Road ahead: 1. We should find out how many use xmlspec, or anything like it, now. Here group chairs coudl help greatly. 2. We should play around with the idea of creating an extension to xmlspec (or write it from the beginning) to represent semantics and intended behaviours (again, please include me in the loop) 3. Any interested party should read the QA Specification Guidelines document, comments are very much appreciated. 4. The topic of whether the HTML version found on the web or the (generated) XML version is the normative one should perhaps be made again, as following the idea that originating this thread will certainly mean that the XML version takes precedence over any other version (when it comes to specification interpretation) 5. Find a relevant forum for discussing this. the QA IG list has been proposed, and I think it may be the right place for it, at least for now. My two (Euro) cents, anyhow. Best, /Dimitris On Friday, May 10, 2002, at 01:40 PM, scott_boag@us.ibm.com wrote: In my experience the human-readability of a spec is an important factor in its successful uptake by the community. I don't think it's totally either one or the other. And most specs are better anyway if they are declarative, in my experience. Also, there is the path that Schema took, which is to provide a tutorial. I'm not sure it merits immediate adoption. It depends on what baby steps may be taken. I believe a large amount of work lies in wait for the creation of test suites for XPath 2.0/XQuery 1.0/XSLT 2.0. We may be able to save a lot of person hours by taking some simple markup steps now. -scott Tim Bray < cc: Jonathan Robie <, Sent by: spec-prod@w3.org, w3c-query-editors@w3.org, www-qa@w3.org, (bcc: Scott w3c-query-editors-req Boag/Cambridge/IBM) uest@w3.org Subject: Re: Testable assertion tagging for W3C specifications 05/09/2002 01:39 PM scott_boag@us.ibm.com wrote: Too often we think of the specification as prose. What it really must be is This is a strong assertion that may be true but probably requires some supporting evidence. In my experience the human-readability of a spec is an important factor in its successful uptake by the community. XML 1.0 is admittedly a counter-example, but should I ever undertake another large-scale core-technology spec editing assignment, I'd put a lot more energy into the prose. Having said all that, the testable-assertions hypothesis probably merits serious investigation. I'm not sure it merits immediate adoption. -Tim