(unknown charset) Re: FYI : a new draft of "QA Framework - Specifications Guidelines" published

On 3 Sep 2002, Dominique [ISO-8859-1] Hazaël-Massieux wrote:

> > Virtually all specifications that define behavior based on inputs are
> > not testable according to the above definition. There is no finite
> > cost-effective process to enumerate all valid inputs. Yet such an
> > enumeration would be required to check that implementation meets the
> > requirement. The guidelines should either limit their scope to
> > non-behavioral specifications or relax the definition of testability.
>
> I'm not sure about this: why should we relax this definition? We're
> just asserting that a behavioral specification is not testable in
> our definition. We're not saying that it's bad, nor that our
> guidelines apply or not to it...

You are not saying, you may be implying. If that's what you want to
say then:

	(a) I would suggest that the statement is made explicit:
	    "behavioral specifications (e.g., SOAP, SMIL?, HTTP) are
	    defined to be not testable"

	(b) I am surprised WG wants to spend so much resources on
	    something that immediately alienates future (and
	    current) behavioral specifications

	(c) I would recommend relaxing the definition so that
	    the resources spent on producing SpecGL are better
	    utilized (as they would apply to more specs). Or does
	    QAWG plan to write BehaveGL next?

> > 	"Checkpoint 1.2. Include User Scenarios. [Priority 2]"
> > 	"Checkpoint 1.3. Provide examples. [Priority 1]"
> >
> > These should have lower priority (priority 3?), IMO. Let specs
> > concentrate on defining the standard. Let text books, articles, and
> > such provide illustrations and examples. Ideal specs have
> > non-normative statements only to make the text readable/understandable
> > by smart humans, IMO.
>
> Do you have examples of specification where examples lead to
> confusion?

While I do not believe it is important to have such examples, yes, I
do. HTTP has many examples that led to incompatible implementations
(ones that followed the normative text were incompatible with the ones
that followed the informal example). One specific case I was involved
with had an informal example that appeared valid to the RFC editors
but could be interpreted differently.

Moreover, I dare to say that a large portion of HTTP spec violations
arise from people implementing by-example rather than by-norm. And
this is _not_ limited to cases where examples are imprecise or wrong.
Even when the example is perfectly fine, people often tend to
implement just what the example covers and not other normative cases.

> As a counter-example, XML Schema for instance has often been
> reproached to be too scarce in examples and hence, hard to
> understand and implement.

Using your logic I should ask whether you have examples where XML
Schema implementations were not compliant because they followed the
norm rather than [absent] examples. Or, better, I should ask whether
you have examples where providing lots of XML Schema examples within
the specs produced more compliant implementations.

But I will not ask these questions because it is the core principle
that is wrong. It is always possible to find examples and
counter-examples.

If XML Schema is so hard to understand and implement because of the
lack of examples, the WG can publish a 'how to implement XML Schema'
tutorial with pretty pictures and examples.

> Well, if the spec writers can't even themselves write examples
> non-conflicting with the specification, there is probably a bigger
> problem in the said specification.

Yes, that bigger problem is on a meta (human) level. Some examples are
conflicting (there are always bugs) and most examples are limited (so
you get incomplete implementations). Humans are bad at manually
duplicating information and should be prohibited from doing so.
Example is a manual and partial duplicate of the normative text.

> > Regardless of what the spec says about such
> > conflicts and their resolution, they do lead to implementation bugs
> > because some developers will look at the example and ignore or
> > misinterpret the normative language.
>
> It's not by making a specification harder to understand that you'll get
> better implementations IMHO. I strongly disagree that a hard to
> understand implementation will imply that only "smart" people will
> implement it, but in the contrary, that probably only the so called
> smart people will implement it correctly.

I did not say that specifications should be made harder to understand!

IMO, specifications should be made precise and succinct, with no
duplication of information, to the extent possible.  Pretty
illustrations do not belong to a spec unless they are normative
themselves. If you assert that it is impossible to write good specs
without plenty of examples, I disagree (but I do not assert that every
spec without examples is good).

> Hmm... A test assertion is "A set of premises that are known to be
> true by definition in the spec." [2]; you don't have to give an
> example for everything that the BNF covers, just give one example of
> what the BNF covers.

Perhaps you could make that intent explicit? Once you do, we can see
wether one example can cover several assertions, etc. In other words,
please define what it means to "cover a test assertion", how many
"examples" are enough, etc. This will make checkpoints testable. They
are testable not in their current shape.

> I think I agree that we might have made too much about the
> dimensions of variability and we should try to re-factor some of it.
> I don't agree though that we should not try to say what is good or
> bad for interoperability since that's part of our goals.

You can advice that less variability is good, but I see no practical
benefit from talking at length about some well-known forms of
variability. You should, IMO, try to make SpecGL have the same
characteristics SpecGL demands from other specs. I believe that absence
of lengthy rhetoric with little practical outcome should be one of
them. If degrees of variability are an interesting issue to talk
about, QAWG can publish an article on the subject.

> I think any judgment checkpoint (that is a checkpoint asserting what
> is good or bad) should have a restriction saying "... or document
> why this doesn't fit your needs".

Ideally, there should be no judgment checkpoints. If something MUST be
documented, demand that it is documented. Period. Leave the rest for
the authors to argue about.

> > 	"Guideline 6. Clarify the relation between deprecated features
> > 	and conformance."
> >
> > This guideline talks about one specific problem that some specs might
> > encounter. I suggest that this guideline is removed because the other
> > guidelines and checkpoints already cover what this guideline,
> > essentially, talks about: the spec must define conformance. A good
> > definition will cover deprecated features as needed, of course.
>
> I disagree. If we follow this road we could just make a one paragraph
> document saying "make a good specification", that would cover
> everything, wouldn't it?

This is a lame argument. If we follow the same road in the opposite
direction, then SpecGL would become a 500-page book talking at length
about various problems that might occur, but mostly irrelevant to any
given spec. The title of that book would be "How to make a good
specification and other stories".

> Deprecated features have appeared as being a tough points in a
> specification evolution, so I think it's important to document it.

No need to document something covered by other checkpoints.

> > 	- Define compliant behavior for every conforming
> > 	  object that the spec is using
> > 	  (no "implicity", good "input")
> >
> > 	- Define compliant behavior for every non-conforming
> > 	  object that an implementation may encounter
> > 	  (no "implicity", bad "input")
>
> I like this way of dividing conformance guidelines. I think that
> detailing what we mean behind that is good too, though.

What you mean must be well explained, yes. There is no reason, and in
fact is often harmful, to go beyond that. For example, there is no
reason to define N out of M (N << M) dimensions of variability if they
are not really needed to define good specs using the approach above.

> Maybe this could be a hint on a reorganization of the dimensions of
> variability guidelines?

Yes, and a very clear hint: remove most of the text that discusses
dimensions of variability. The text overall usefulness to spec authors
is highly questionable. I cannot see how a future spec author would
write a better spec if the text remains in SpecGL because everything
it talks about is (or should be) covered by more abstract/general
checkpoints.

> > 	"Checkpoint 9.5. If extensions are allowed, register or publish
> > 	them. [Priority 3]"
> >
> > In general, this is totally unnecessary and introduces too much overhead
> > provided the good/bad input guidelines above are satisfied.
>
> Have you some examples to provide? Namely examples of an extension
> mechanism without registration that didn't harm interoperability and
> examples of an extension with registration that was too much overhead?

Note that most existing specs do not conform to SpecGL and do not
satisfy my "provided ..." clause above.

But if you must have an example for everything, I can say that
extension headers in HTTP/1.1 worked quite well without any
registration (as long as implementations dealing with them were
compliant in that regard). There has recently been an effort to
register/collect known extension headers, but the protocol would be
impossible to use in practice if such registration was mandatory
before anybody can start using those headers.

> I have at least one counter-example of your assertion:
> - XSLT provides a clearly defined extension mechanism that didn't prove
> useful until some people decided to register a set of them so that you
> get a good enough interoperability (see
> http://www.xml.com/pub/a/2001/02/14/deviant.html and
> http://www.exslt.org/)

To provide a counter example to an "In general, ..." assertion, you
have to prove that there are no valid examples. One or N counter
examples could be simply exceptions to a general rule.

I think you are missing the important point behind minor details
though. If you demand that extensions are registered and published,
they are no longer extensions! They essentially become a part of the
ever-evolving spec. By its very definition, extension is something
that can be added without being known to others and still work. That's
what important.

> > This is good but probably impractical. Even W3C "valid *ML" images do
> > not (and cannot) satisfy this verbose format.
>
> Being a priority 3 checkpoint, I don't think that's a big issue if it's
> not practical.

Great. I will ignore all priority 3 checkpoints from now on. Why waste
time on something that is impractical? Are we trying to maximize the
length of SpecGL?

Seems to me that SpecGL violates its own recommendation for reducing
variability by adding "optional" checkpoints with no practical value.

> > 	"Guideline 12. Publish an Implementation Conformance Statement
> > 	proforma."
> >
> > This is useful in some cases, but has nothing to do with the spec
> > itself, IMO. I would remove this guideline.
>
> As you have noticed, the "specification" GL are not just about the
> specification understood as a document, but about the abstract
> specification as Al Gilman described it [3]. In this meaning, I
> think it's part of this document.

I do not understand the scope of SpecGL then. My mental abilities are
insufficient to extract a definition of an "abstract specification"
from [3]. Perhaps SpecGL should contain a clear definition of its
scope? "SpecGL covers specification understood as a document and
specifications as Al Gilman described it" seems vague to me.

Hmm... Looks like SpecGL does not define its scope at all, violating
its own recommendation (Checkpoint 1.1)! Did I miss it?

> > 	"Finally, using an advanced schema for specification authoring
> > 	allows for greater control over coverage of the specification in
> > 	the Test Suite."
> >
> > Not really. It may make greater control over coverage easier to
> > implement, that's all. Plain ASCII text allows for the same coverage
> > control as *ML.
>
> Sure, but I'm not sure that the distinction is that much relevant.

It is not a distinction. The "Finally, " statement quoted above is
simply wrong/false. I hope that is relevant.

> > 	"Test assertions (that are part of a specification) are publicly
> > 	 reviewed and agreed to"
> >
> > So can be test assertions published outside of the spec.
>
> The thing is that by binding them to the specification, you make them
> normative and you make them reviewed by the various proponents of the
> W3C Process. Maybe we should clarify the rationale?

See below for why binding and self-review is bad.

> > Moreover, I can make an argument that including test assertions in the
> > spec may cripple future test suites and even implementations!  Spec
> > implementors will see the assertions and make sure they write code that
> > passes those test cases. Test suite developers will implement the same
> > assertions that the implementors were looking at (at least they will
> > start with them, many will not go beyond). As a result, the
> > effectiveness of a test suite will be minimal. It is a well known rule
> > of thumb that best testing is done by non-implementors. Including
> > "standard" test assertions violates this rule in practice (in theory it
> > would not matter, of course).
>
> Do you have examples to illustrate this?

It is a well known fact that testing is more effective if done by
non-implementors. I am not sure what kind of examples you are asking
for. Would the fact that most software implementations out there
violate the protocols they implement suffice?


Thank you,

Alex.

> 3. http://lists.w3.org/Archives/Public/www-qa/2002Aug/0031.html

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Thursday, 5 September 2002 12:50:06 UTC