Re: A Method for Writing Testable Conformance Clauses and its Applications (Was Re: Write up of test assertion extraction methodology)

On Jan 20, 2010, at 6:03 AM, Doug Schepers <schepers@w3.org> wrote:

> Hi, Marcos-
>
> Marcos Caceres wrote (on 1/19/10 10:49 AM):
>> Hi all,
>> A draft of "A Method for Writing Testable Conformance Clauses and its
>> Applications" in now available for review online [1]. For those that
>> have not seen it, it basically just documents how we are  
>> standardizing
>> the Widget specs and some basic QA processes:
>>
>> http://dev.w3.org/2008/dev-ind-testing/extracting-test-assertions-pub.html
>>
>> Please consider this a working draft, as it likely contains typos,  
>> and a
>> couple of half-baked ideas, etc. Comments are, of course, welcomed.  
>> It
>> is expected that this document will be published as a working group  
>> note
>> at some point in the future.
>
> This is an interesting doc and case study, with useful best- 
> practices that mirror some of my own experience in creating specs  
> and test suites.  I plan on integrating this into specs I edit,  
> where possible.
>
> I think you could emphasize algorithms a bit more, which is  
> traditionally a weak point in W3C specs.

Can you elaborate a bit here. I'm really interested because I've  
recieved both positive and negative feedback about algoriths. E.g.  
Some say it's great to have them as step by step; others says it's  
overly prescriptive and takes the freedom/fun/art out of implementing.

>
> It may or may not be appropriate for this document to discuss  
> document readability more; W3C specs traditionally serve multiple  
> audiences, such as implementers, tutorial/book writers, and content  
> creators (authors), and while there is a move to make specs more  
> implementer-focused and separate out these concerns, this decreases  
> the value of the specs as community artifacts, makes tutorials more  
> likely to be available only much later and to be error-prone, since  
> they are not written by the same group of people who wrote the prose  
> of the spec.

Agree completely; you've exposed my political agenda:)  I strongly  
believe specs should be legible and usable by all its community of  
users. I don't believe "implementers" should be priveledged and that  
they have same magical ability to read specs that we mere mortals  
lack. I believe specs should be written as accesible as possible in  
their language and structure.

>
> I really like the goal and execution of this practice.  Some of it  
> reminds me of my experimental Speki [1] project, which was an  
> exploration of a wiki toolbar to mark up specs for easier spec  
> production.   One aspect of this was to create special stylesheets  
> for making the conformance criteria more obvious (follow the link,  
> and select the "implementers" presentation on the sidebar for a  
> demonstration).  I applied some of this markup to DOM3 Events, but  
> not consistently enough.  This was part of my Project Tinker [2]  
> architecture proposal (which I've neglected for a year or two).

I'll be sure to take a look.

>
>
> Overall, I am very supportive of this kind of detailed analysis, and  
> I appreciate the time and care you took to develop and write it up.

Thanks!

> This kind of document stands to make the production and quality of  
> specs much better.  My chief suggestion is that, having developed  
> your workflow and tools, you should make them available to other  
> groups;

I've been talking to Robin about adding what is relevant to ReSpec. We  
have not yet reached any agreement, however.

> I am interested in reusing your tools for DOM3 Events and SVG,  
> specifically the markup and implementation report tools, and with  
> some modification, the test suite markup and tools.

Great!

>  SVG has its own tools and processes for some parts of this, but I  
> strongly believe that the benefit increases the more specs use the  
> same toolchain.

Agreed.

>
>
> Here are some specific comments, mostly typo corrections or  
> clarification suggestions:

I think Dom has now integrated your suggestions/fixes.

>
>
> 1. Common mistakes
>
> * For each of the commons mistakes, include corrections for the  
> faulty prose.
>
>
> 2. The method
>
> * "stable identifier" may be a little vague and abstract; if you  
> mean to add an @id, say, "give them a stable identifier, (such as)  
> an 'id' attribute value that is consistent between drafts".  If you  
> mean something else, describe it.
>
> * Typo: "(see )"
>
> * Typo: "learnt" -> "learned"
>
>
> 2.1 Relationship to the standardization process
>
> * "tangible objects": it might be better to use the term  
> "deliverables", since that is the wording of the Process Document  
> and charters; at the least, associate the 2 terms... "tangible  
> objects (also called the "deliverables" of the group) ..

>
> * Include "issue tracking software" in the list of tools provided by  
> the W3C, which can be used as part of the test/spec feedback loop.
>
>
> 3. Structural components of a conformance clause
>
> * "Product" might be better stated as a "Conformance Category",  
> which includes authoring tools and authors.  While tests against  
> authored content are not in the scope for this document, editors  
> need to be aware of the various conformance categories that they  
> need to keep in mind when writing, of which "user agents" are only  
> one category.  DOM3 Events describes the following conformance  
> categories: Web browsers and other dynamic or interactive user  
> agents; Authoring tools; Authors and content; and Specifications and  
> host languages.
>
>
> 4. Conventions for marking-up conformance clauses
>
> * Typo: "anylising" -> "analyzing"
>
> * Typo: "Prerequisites: <"If"" -> "Prerequisites: "If""
>
> * Typo: "Working Groups is" -> "Working Group is"
>
> * "Hyperlink to the appropriate terms."  Show, don't tell.  Give us  
> an example here, to contextualize what you mean by "the appropriate  
> terms", even though the full example is down below: "Hyperlink to  
> the appropriate terms. For example, "encounters a <a class="term"  
> href="#file">file</a> matching""
>
>
> 5. Extracting conformance clauses
>
> * "in favor of using JavaScript" -> "in favor of using a custom  
> JavaScript library called _____"... make it easy to discuss and share
>
> * Typo: "with am XML document" -> "with an XML document"
>
> * Typo: "it allows to quickly assess" -> "it allows quick assessment"
>
> * Typo: "informant ion" -> "information"
>
> * Typo: "describe individual results" -> "describes individual  
> results"
>
> * Typo: "attrite" -> "attribute"
>
> * Grammaro: This is a sentence fragment: "And where the test ran,  
> but it was not possible to determine if the test actually passed or  
> failed (labeled as "incomplete")." -> "And where the test ran, but  
> it was not possible to determine if the test actually passed or  
> failed, the verdict was indicated as "incomplete"."
>
> * Typo: "three separate steps (, marking it up the specification," - 
> > "three separate steps (marking it up in the specification," or  
> "three separate steps (marking up the specification,"
>
> * Typo: "and in practice to work best as an iterative process." ->  
> "and in practice work best as an iterative process."
>
>
> [1] http://www.schepers.cc/speki/index.php?title=MonkeyML
> [2] http://www.schepers.cc/tinker/
>
> Regards-
> -Doug Schepers
> W3C Team Contact, SVG and WebApps WGs

Thanks again for the detailed review and you supportive comments.
>

Received on Friday, 22 January 2010 05:05:13 UTC