Re: A Method for Writing Testable Conformance Clauses and its Applications (Was Re: Write up of test assertion extraction methodology)

Hi, Marcos-

Marcos Caceres wrote (on 1/19/10 10:49 AM):
> Hi all,
> A draft of "A Method for Writing Testable Conformance Clauses and its
> Applications" in now available for review online [1]. For those that
> have not seen it, it basically just documents how we are standardizing
> the Widget specs and some basic QA processes:
>
> http://dev.w3.org/2008/dev-ind-testing/extracting-test-assertions-pub.html
>
> Please consider this a working draft, as it likely contains typos, and a
> couple of half-baked ideas, etc. Comments are, of course, welcomed. It
> is expected that this document will be published as a working group note
> at some point in the future.

This is an interesting doc and case study, with useful best-practices 
that mirror some of my own experience in creating specs and test suites. 
  I plan on integrating this into specs I edit, where possible.

I think you could emphasize algorithms a bit more, which is 
traditionally a weak point in W3C specs.

It may or may not be appropriate for this document to discuss document 
readability more; W3C specs traditionally serve multiple audiences, such 
as implementers, tutorial/book writers, and content creators (authors), 
and while there is a move to make specs more implementer-focused and 
separate out these concerns, this decreases the value of the specs as 
community artifacts, makes tutorials more likely to be available only 
much later and to be error-prone, since they are not written by the same 
group of people who wrote the prose of the spec.

I really like the goal and execution of this practice.  Some of it 
reminds me of my experimental Speki [1] project, which was an 
exploration of a wiki toolbar to mark up specs for easier spec 
production.   One aspect of this was to create special stylesheets for 
making the conformance criteria more obvious (follow the link, and 
select the "implementers" presentation on the sidebar for a 
demonstration).  I applied some of this markup to DOM3 Events, but not 
consistently enough.  This was part of my Project Tinker [2] 
architecture proposal (which I've neglected for a year or two).


Overall, I am very supportive of this kind of detailed analysis, and I 
appreciate the time and care you took to develop and write it up.  This 
kind of document stands to make the production and quality of specs much 
better.  My chief suggestion is that, having developed your workflow and 
tools, you should make them available to other groups; I am interested 
in reusing your tools for DOM3 Events and SVG, specifically the markup 
and implementation report tools, and with some modification, the test 
suite markup and tools.  SVG has its own tools and processes for some 
parts of this, but I strongly believe that the benefit increases the 
more specs use the same toolchain.


Here are some specific comments, mostly typo corrections or 
clarification suggestions:


1. Common mistakes

* For each of the commons mistakes, include corrections for the faulty 
prose.


2. The method

* "stable identifier" may be a little vague and abstract; if you mean to 
add an @id, say, "give them a stable identifier, (such as) an 'id' 
attribute value that is consistent between drafts".  If you mean 
something else, describe it.

* Typo: "(see )"

* Typo: "learnt" -> "learned"


2.1 Relationship to the standardization process

* "tangible objects": it might be better to use the term "deliverables", 
since that is the wording of the Process Document and charters; at the 
least, associate the 2 terms... "tangible objects (also called the 
"deliverables" of the group) ..."

* Include "issue tracking software" in the list of tools provided by the 
W3C, which can be used as part of the test/spec feedback loop.


3. Structural components of a conformance clause

* "Product" might be better stated as a "Conformance Category", which 
includes authoring tools and authors.  While tests against authored 
content are not in the scope for this document, editors need to be aware 
of the various conformance categories that they need to keep in mind 
when writing, of which "user agents" are only one category.  DOM3 Events 
describes the following conformance categories: Web browsers and other 
dynamic or interactive user agents; Authoring tools; Authors and 
content; and Specifications and host languages.


4. Conventions for marking-up conformance clauses

* Typo: "anylising" -> "analyzing"

* Typo: "Prerequisites: <"If"" -> "Prerequisites: "If""

* Typo: "Working Groups is" -> "Working Group is"

* "Hyperlink to the appropriate terms."  Show, don't tell.  Give us an 
example here, to contextualize what you mean by "the appropriate terms", 
even though the full example is down below: "Hyperlink to the 
appropriate terms. For example, "encounters a <a class="term" 
href="#file">file</a> matching""


5. Extracting conformance clauses

* "in favor of using JavaScript" -> "in favor of using a custom 
JavaScript library called _____"... make it easy to discuss and share

* Typo: "with am XML document" -> "with an XML document"

* Typo: "it allows to quickly assess" -> "it allows quick assessment"

* Typo: "informant ion" -> "information"

* Typo: "describe individual results" -> "describes individual results"

* Typo: "attrite" -> "attribute"

* Grammaro: This is a sentence fragment: "And where the test ran, but it 
was not possible to determine if the test actually passed or failed 
(labeled as "incomplete")." -> "And where the test ran, but it was not 
possible to determine if the test actually passed or failed, the verdict 
was indicated as "incomplete"."

* Typo: "three separate steps (, marking it up the specification," -> 
"three separate steps (marking it up in the specification," or "three 
separate steps (marking up the specification,"

* Typo: "and in practice to work best as an iterative process." -> "and 
in practice work best as an iterative process."


[1] http://www.schepers.cc/speki/index.php?title=MonkeyML
[2] http://www.schepers.cc/tinker/

Regards-
-Doug Schepers
W3C Team Contact, SVG and WebApps WGs

Received on Tuesday, 19 January 2010 22:03:56 UTC