Re: FYI : a new draft of "QA Framework - Specifications Guidelines" published

[sorry for the rather long mail]

Le ven 30/08/2002 à 17:06, Alex Rousskov a écrit :
> Hi there,

Hi, and thanks for your input!
 
 
> 	"1.4 Terminology The keywords "MUST", "MUST NOT", ... will be
> 	 used as defined in RFC 2119"
> 
> The document does not use MUST, SHOULD, or MAY keywords. Or, at least,
> these words are not used in the way suggested by RFC 2119 (capitalized
> to make them look like keywords).

True, I think we'll need to clarify this indeed (we have 3 non
capitalized shall, 30 should, 16 required, 37 must, 7 optional, 5
recommended and 73 may according to [1]).
 
> 	"A specification is testable if there exists some finite
> 	cost-effective process with which a person or software can check
> 	that the requirement has been met."
> 
> Virtually all specifications that define behavior based on inputs are
> not testable according to the above definition. There is no finite
> cost-effective process to enumerate all valid inputs. Yet such an
> enumeration would be required to check that implementation meets the
> requirement. The guidelines should either limit their scope to
> non-behavioral specifications or relax the definition of testability.

I'm not sure about this: why should we relax this definition? We're just
asserting that a behavioral specification is not testable in our
definition. We're not saying that it's bad, nor that our guidelines
apply or not to it...
 
> 	"Guideline 1. Define Use Cases."
> 
> I suggest that this is renamed to "Define Scope". Use cases often have a
> damaging effect because they limit the way spec authors interpret their
> spec (but not the way spec readers do). Scope is important and can be
> normative. Use cases are complimentary and should be used for
> illustrative (not normative) purposes only, if at all.

Defining the scope is the first checkpoint. I think that "use cases"
describe quite well the various checkpoints under this guideline.

> 	"Checkpoint 1.1. Define the scope of the specification. ... The
> 	specification must clearly define what scenarios are in scope
> 	and what are out of scope in order to be interpreted,
> 	implemented, and tested correctly."
> 
> Replace "clearly define" with "define" to make this checkpoint testable.
> In general, avoid adjectives in requirements as they have no value and
> may do harm.

Good point.

> 	"Checkpoint 1.2. Include User Scenarios. [Priority 2]"
> 	"Checkpoint 1.3. Provide examples. [Priority 1]"
> 
> These should have lower priority (priority 3?), IMO. Let specs
> concentrate on defining the standard. Let text books, articles, and
> such provide illustrations and examples. Ideal specs have
> non-normative statements only to make the text readable/understandable
> by smart humans, IMO.

Do you have examples of specification where examples lead to confusion?
As a counter-example, XML Schema for instance has often been reproached
to be too scarce in examples and hence, hard to understand and
implement.
 
> 	"The more numerous the examples, the better it is for the
> 	specifications comprehensibility."
> 
> I strongly disagree. The more non-normative scenarios/examples there
> are, the higher the chance of a conflict between a normative statement
> and a non-normative example.

Well, if the spec writers can't even themselves write examples
non-conflicting with the specification, there is probably a bigger
problem in the said specification.

> Regardless of what the spec says about such
> conflicts and their resolution, they do lead to implementation bugs
> because some developers will look at the example and ignore or
> misinterpret the normative language.

It's not by making a specification harder to understand that you'll get
better implementations IMHO. I strongly disagree that a hard to
understand implementation will imply that only "smart" people will
implement it, but in the contrary, that probably only the so called
smart people will implement it correctly.
 
> 	"Checkpoint 1.4. Ensure that every test assertion is covered by
> 	an example. [Priority 3]"
> 
> Why?! Satisfying this requirement will make protocol and similar specs
> huge without any added benefit. If, for example, a BNF specifies the
> protocol header syntax, why would anybody want to generate an example
> for everything that BNF covers?!

Hmm... A test assertion is "A set of premises that are known to be true
by definition in the spec." [2]; you don't have to give an example for
everything that the BNF covers, just give one example of what the BNF
covers. 

>          "G 2. Identify what needs to conform and how.
>          G 3. Address the use of profiles to divide the technology.
>          G 4. Address the use of modules to divide the technology."
> 
> I would shorten the three guidelines above into a single guideline with
> 3-5 paragraph of text addressing a single (and the only, IMO) important
> issue: "G 2. Define conformance".  That is, the spec MUST define what it
> means to conform to the spec. Given a device/document/software within
> the scope of the spec, it SHOULD be possible to test for conformance.
> That's all.
> 
> There is no use for SpecGL to enumerate possible products that might
> be in scope of future specs. There is no use to talk at length about
> profiles and modules, their benefits and side-effects. Leave these
> arguments for the spec authors. You are not going to provide a clear
> one-size-fits-all answer any way. Demand only what is essential and
> universal: a conformance clause.

I think I agree that we might have made too much about the dimensions of
variability and we should try to re-factor some of it. I don't agree
though that we should not try to say what is good or bad for
interoperability since that's part of our goals.

I think any judgment checkpoint (that is a checkpoint asserting what is
good or bad) should have a restriction saying "... or document why this
doesn't fit your needs".
 
> 	"Guideline 6. Clarify the relation between deprecated features
> 	and conformance."
> 
> This guideline talks about one specific problem that some specs might
> encounter. I suggest that this guideline is removed because the other
> guidelines and checkpoints already cover what this guideline,
> essentially, talks about: the spec must define conformance. A good
> definition will cover deprecated features as needed, of course.

I disagree. If we follow this road we could just make a one paragraph
document saying "make a good specification", that would cover
everything, wouldn't it? Deprecated features have appeared as being a
tough points in a specification evolution, so I think it's important to
document it.

> 	"Checkpoint 8.4. Include a statement regarding consistent
> 	handling of a discretionary item within an implementation.
> 	[Priority 2]
> 
>         The effect of each individual discretionary item should be
> 	consistent within a single implementation. For example, a
> 	browser's rendering of a XSL-FO (XSL Formatting Object) should
> 	be the same for every invocation regardless of the document
> 	instance."
> 
> Why on Earth would these guidelines require something specific like
> that? Why handling of a discretionary item within an implementation
> should be consistent? Care to define an implementation in this context?
> Is browser X that is user-configured to do FOO the same implementation
> as browser X that is user-configured to do BAR? Care to define an
> invocation in this context? Can browser display cached documents
> differently from first-hand responses? Etc., etc. This is just too
> low-level to provide a general guideline, IMO.

The idea is to lower the bad effect of discretionary items on a
specification. But it would probably need some rewording or
clarifications.
 
> 	"Guideline 9. Allow extensions or NOT!"
> 
> This guideline sounds cool, but should be removed because it does not
> cover any new ground, IMO: If the spec explicitly mentions an extension,
> it MUST document compliant behavior just like for any other
> non-extension, due to other guidelines. The problem arises only when the
> spec implicitly allows something without defining a conformant reaction
> to that something. That problem should be addressed by more general
> guidelines:
> 
> 	- Define compliant behavior for every conforming
> 	  object that the spec is using
> 	  (no "implicity", good "input")
> 
> 	- Define compliant behavior for every non-conforming
> 	  object that an implementation may encounter
> 	  (no "implicity", bad "input")

I like this way of dividing conformance guidelines. I think that
detailing what we mean behind that is good too, though. Maybe this could
be a hint on a reorganization of the dimensions of variability
guidelines?

> 	"Checkpoint 9.5. If extensions are allowed, register or publish
> 	them. [Priority 3]"
> 
> In general, this is totally unnecessary and introduces too much overhead
> provided the good/bad input guidelines above are satisfied.

Have you some examples to provide? Namely examples of an extension
mechanism without registration that didn't harm interoperability and
examples of an extension with registration that was too much overhead?

I have at least one counter-example of your assertion:
- XSLT provides a clearly defined extension mechanism that didn't prove
useful until some people decided to register a set of them so that you
get a good enough interoperability (see
http://www.xml.com/pub/a/2001/02/14/deviant.html and
http://www.exslt.org/)

> 	"Checkpoint 10.5. Identify all dimensions of variability that
> 	are not used. [Priority 1]
> 
> I think this is an unnecessary overkill. The scope and conformance
> sections MUST define scope and conformance. We should not require
> authors to define non-scope and non-conformance as well. Everything that
> is not covered by the scope, is out of scope. Everything that does not
> conform, does not. What good does it make to state *in the conformance
> clause* that, say, ``extensions are [not] allowed''?! The reader still
> has to read the specs to understand what an extension is anyway. While
> doing so, the reader will encounter specific conformance requirements
> for those extensions, if any.

Agreed. I think we should not make the conformance section too verbose. 

 
> 	"Checkpoint 11.2. Provide specific wording of the claim.
> 	[Priority 3]
> 
>         A well-formed conformance claim includes: date of the claim,
> 	specification name, date and version, URI of the specification,
> 	conformance level satisfied, and information about the subject
> 	(that which is claiming conformance). Information about the
> 	subject refers to information such as, the name of the software
> 	or software component, version information, and operating
> 	environment."
> 
> This is good but probably impractical. Even W3C "valid *ML" images do
> not (and cannot) satisfy this verbose format.

Being a priority 3 checkpoint, I don't think that's a big issue if it's
not practical.

> 	"Guideline 12. Publish an Implementation Conformance Statement
> 	proforma."
> 
> This is useful in some cases, but has nothing to do with the spec
> itself, IMO. I would remove this guideline.

As you have noticed, the "specification" GL are not just about the
specification understood as a document, but about the abstract
specification as Al Gilman described it [3]. In this meaning, I think
it's part of this document.
 
> 	"Checkpoint 13.2. Distinguish normative and informative text.
> 	[Priority 2]"
> 
> This should have the highest priority (Priority 1?), IMO. The spec is no
> good if normative and informative statements are indistinguishable.

Agreed.
 
> 	"Finally, using an advanced schema for specification authoring
> 	allows for greater control over coverage of the specification in
> 	the Test Suite."
> 
> Not really. It may make greater control over coverage easier to
> implement, that's all. Plain ASCII text allows for the same coverage
> control as *ML.

Sure, but I'm not sure that the distinction is that much relevant.

> 	"Guideline 15. Include test assertions."
> 
> The rationale the
> guidelines give for including Guideline 15 are not really specific to
> the inclusion of test assertions into the spec:
> 
> 	"Including test assertions as part of the specification
> 	facilitates and promotes the development of test materials."
> 
> So lets include everything that facilitates and promotes then. Let's
> include translations to other languages, alternative formats, author
> commentary, code samples, etc. etc.
> 
> 	"Test assertions (that are part of a specification) are publicly
> 	 reviewed and agreed to"
> 
> So can be test assertions published outside of the spec.

The thing is that by binding them to the specification, you make them
normative and you make them reviewed by the various proponents of the
W3C Process. Maybe we should clarify the rationale?

 
> Moreover, I can make an argument that including test assertions in the
> spec may cripple future test suites and even implementations!  Spec
> implementors will see the assertions and make sure they write code that
> passes those test cases. Test suite developers will implement the same
> assertions that the implementors were looking at (at least they will
> start with them, many will not go beyond). As a result, the
> effectiveness of a test suite will be minimal. It is a well known rule
> of thumb that best testing is done by non-implementors. Including
> "standard" test assertions violates this rule in practice (in theory it
> would not matter, of course).

Do you have examples to illustrate this?


> 	"The test assertions of this Specification Guidelines document
> 	are found in the prioritized checkpoints. A checkpoint will
> 	contain at least one, and may contain multiple individual
> 	requirements. These requirements are the test assertions of this
> 	specification."
> 
> Note that most of what you call "test assertions" cannot be tested with
> because they contain undefined adjectives like "clear".

Agreed. If you have noticed other type of misuses, let us know.

> 	"test assertion: any sentence or equivalent assemblage of
> 	 words and phrases that prescribes the behavior that must be
> 	 obtained when a stimulus occurs under a certain set of
> 	 conditions. (See also QA Glossary [QA-GLOSSARY].)"
> 
> Can a test assertion be expressed in a formal language without words
> and phrases rather than in sentences of a human language?

Err... anyway, this definition doesn't match the one used in the
glossary. We need to fix this.
 
Dom

1.
http://www.w3.org/2002/08/diction?uri=http%3A%2F%2Fwww.w3.org%2FTR%2F2002%2FWD-qaframe-spec-20020826%2F
2. http://www.w3.org/QA/glossary
3. http://lists.w3.org/Archives/Public/www-qa/2002Aug/0031.html
-- 
Dominique Hazaël-Massieux - http://www.w3.org/People/Dom/
W3C/INRIA
mailto:dom@w3.org

Received on Tuesday, 3 September 2002 10:12:18 UTC