Re: Testable assertion tagging for W3C specifications

Sorry for jumping in on the thread at so late a stage.

Being very interested in the topic and also being the principal author 
of the relevant parts of the Specification Guidelines document (which I 
co-edit), let me first say that I look forward to the results this 
discussion will lead to. Please include me in any relevant messaging, 
even off these lists.

A word of warning; this is a rather long email, as I try to reply to and 
discuss as much as possible (of course, contrary to any implicit rule 
out there).

General outline (in reply to Scott's original posting, parts of which 
are pasted, my comments prefixed by [dd]):

---
GOALS:
    Allow an external document (test case, erratum, email, etc.) to point
    directly at a "testable" normative sentence in a Recommendation.

[dd] This would clearly simplify the task of, if we look at tests, 
knowing which part in particualr is being tested, but requires structure 
and issues tracking. This in turn implies that it may need to be an 
intra-W3C "standard".

    Encourage document editors to view some of the sentences as "test
    assertions" and to write them in a style that conveys precisely what
    they declare.

[dd] Any disambiguation is welcome. However, there are issues with, as I 
see it:

1. Forcing specification authors to write in a particular markup (which 
could be worth it, given the complexity they need to bea ble to master 
to begin with to author a spec)
2. Different WG's will certainly have their own views on how the xmlspe, 
for example, should be extended.

    Explore possibilities for machine processing of testable sentences in
    the future.

[dd] This is already being done in an embryo-stage way, more on how we 
generate the DOM Test Suite markup language below. We have had positive 
responses from this approach, as it treats the specification (the XML 
version, actually) more normatively than just serving to produce the 
human readable HTML output.

    Link error assertions to error catalogues (see the work that Mike Kay 
is
    doing with the XSLT document: (
    http://www.w3.org/TR/xslt20/#error-summary)).
    Provide a tagging scheme for testing of grammatical statements, such 
as
    the ad-hoc one employed in the XPath/XQuery specifications.
    Possibly provide markup also for discretionary behavior.

[dd] In general, I think the cost/benefit analysis woul end up with the 
follwing (The QA activity, of course, tries to emphasize the benefits, 
and for this and other reasons it is important that people out there 
raise their voice at an early stage)

Costs

1. "another forced upon us way to write", as has already been discussed 
in this thread. People view the added cost as counter productive, given 
the amount of work they already do.

My personal argument would be to look for the added benefits, which 
include:

a. Easier machine processing of the specification, which allows for 
easier tracking (of tests, errata, issues, and so forth)
b. Easier generation of different types of output (HTML for human 
reading, HTML+SVG for graphical representation, XML for implementations 
specific calls to the spec to do testing on the fly, and so forth)
c. Added degree of normativity to (what I consider to be) the important 
portion of the specification: the XML document. This of course leads to 
digression, but is what taking the idea of enhancing the markup leads to 
eventually

2. Complicates authoring beyond need, even having adopted the extended 
markup. It is tedious to have to use intricate markup to write simple 
things.

a. I don't think we'd ever want to use complicated markup fo write 
simple things, just add to what we do already. In the long run, it would 
certainly be beneficial to be able to let your implementation check 
whether it passes a test and report back to you instead of having to 
read the spec, interpret it, write the (proprietary, since there is no 
agreed on syntax) test, run it, interpret the result, re-reading the 
spec (and re-interpreting it, as we've all seen many times). The added 
benefit is that using intricate markup to write assertions allows for 
easier interpretation of the spec; in particular it greatly simplifies 
testing interdepencies between parts of specifications but also 
interdependent specifications (DOM/HTML, for example)

3. Not all specification aim at specifying the same thing, therefore 
extending a particular markup (which is not used by everyone, in any 
case) may not meet the aim.

Benefits:

1. In general, I think the cost is short term, once adopted it will 
begin to show its advantages, making it more acceptable to new WG's and 
newer version of existing technologies.

2. Having an agreed on markup allows for separating between the 
assertions (where assertion is to be understood as David Marston 
explained earlier), the representation of those assertions (which we now 
view as the specification) and finally mechanisms for asserting tests, 
running them, reporting and so forth. In general, the aim to write 
implementable specifications and making it easier to implement them is 
furthered.

3. Machine calls can be used to ascertain compliance with particular 
specifications, thus lessening the discussion on details in 
specifications done by the same people who wrote it to begin with.

---

DOM Framework:

One example of many is what we did in the DOM TS framework 
(http://www.w3.org/DOM/Test). Here we use XSLT transforms to generate 
the markup language directly from the XML documents that serve to 
produce the HTML output. The TS (in XML) markup mimics code, includes 
pointers to what part of the specification the test tests, and so forth. 
XSLT is used, again, to produce the two official bindings (ECMA and 
Java), but other ones have been produced too (one C++ binding). This 
means that one test case description generates many test instances (one 
per binding). Furthermore, it is straightforward to group tests to form 
specific test suites (DOM allows for different setups, so to speak, for 
example entity handling) so that one can ensure that the implementation 
is being tested on its own merits. Finally, test reporting is greatly 
enhanced.

The benefits we've seen is that the final deliverable (one TS per level 
of the DOM) is made easier, _once_ people come over the threshold of 
using a particular syntax (which speaks for the fact that there is 
resistance to changing tools, so to speak). However, once people saw 
that the new framework was worth supporting, more an more actors 
(re)wrote tests in order to comply with the framework.

Road ahead:

1. We should find out how many use xmlspec, or anything like it, now. 
Here group chairs coudl help greatly.
2. We should play around with the idea of creating an extension to 
xmlspec (or write it from the beginning) to represent semantics and 
intended behaviours (again, please include me in the loop)
3. Any interested party should read the QA Specification Guidelines 
document, comments are very much appreciated.
4. The topic of whether the HTML version found on the web or the 
(generated) XML  version is the normative one should perhaps be made 
again, as following the idea that originating this thread will certainly 
mean that the XML version takes precedence over any other version (when 
it comes to specification interpretation)
5. Find a relevant forum for discussing this. the QA IG list has been 
proposed, and I think it may be the right place for it, at least for now.

My two (Euro) cents, anyhow.

Best,

/Dimitris

On Friday, May 10, 2002, at 01:40 PM, scott_boag@us.ibm.com wrote:

>
>> In my experience the human-readability of a spec
>> is an important factor in its successful uptake by the community.
>
> I don't think it's totally either one or the other.  And most specs are
> better anyway if they are declarative, in my experience.  Also, there is
> the path that Schema took, which is to provide a tutorial.
>
>> I'm not sure it merits immediate adoption.
>
> It depends on what baby steps may be taken.  I believe a large amount of
> work lies in wait for the creation of test suites for XPath 2.0/XQuery
> 1.0/XSLT 2.0.  We may be able to save a lot of person hours by taking 
> some
> simple markup steps now.
>
> -scott
>
>
>
>
>
>                       Tim Bray
>                       <tbray@textuality.com        To:       
> scott_boag@us.ibm.com
>>                            cc:       Jonathan Robie 
>> <jonathan.robie@datadirect-technologies.com>,
>                       Sent by:                      spec-prod@w3.org, 
> w3c-query-editors@w3.org, www-qa@w3.org, (bcc: Scott
>                       w3c-query-editors-req         Boag/Cambridge/IBM)
>                       uest@w3.org                  Subject:  Re: 
> Testable assertion tagging for W3C specifications
>
>
>                       05/09/2002 01:39 PM
>
>
>
>
>
>
> scott_boag@us.ibm.com wrote:
>
>> Too often we think
>> of the specification as prose.  What it really must be is
>
> This is a strong assertion that may be true but probably requires some
> supporting evidence.  In my experience the human-readability of a spec
> is an important factor in its successful uptake by the community.  XML
> 1.0 is admittedly a counter-example, but should I ever undertake another
> large-scale core-technology spec editing assignment, I'd put a lot more
> energy into the prose.
>
> Having said all that, the testable-assertions hypothesis probably merits
> serious investigation.  I'm not sure it merits immediate adoption. -Tim
>
>
>
>
>
>
>