Re: FYI : a new draft of "QA Framework - Specifications Guidelines" published

Hi there,

	I have a few high-level comments on the published
Specifications Guidelines. Overall, I find that the guidelines attempt
to define/restrict/limit a lot more than they can or should. I also
strongly disagree with an apparent intent to make every spec into an
exciting novel to be enjoyed by humans on their way to work. IMHO,
ideal specs should be succinct and formal, with no duplication of
information, to the extent humanly possible.

	More Specific comments are below.


	"1.4 Terminology The keywords "MUST", "MUST NOT", ... will be
	 used as defined in RFC 2119"

The document does not use MUST, SHOULD, or MAY keywords. Or, at least,
these words are not used in the way suggested by RFC 2119 (capitalized
to make them look like keywords).


	"A specification is testable if there exists some finite
	cost-effective process with which a person or software can check
	that the requirement has been met."

Virtually all specifications that define behavior based on inputs are
not testable according to the above definition. There is no finite
cost-effective process to enumerate all valid inputs. Yet such an
enumeration would be required to check that implementation meets the
requirement. The guidelines should either limit their scope to
non-behavioral specifications or relax the definition of testability.


	"Guideline 1. Define Use Cases."

I suggest that this is renamed to "Define Scope". Use cases often have a
damaging effect because they limit the way spec authors interpret their
spec (but not the way spec readers do). Scope is important and can be
normative. Use cases are complimentary and should be used for
illustrative (not normative) purposes only, if at all.


	"Checkpoint 1.1. Define the scope of the specification. ... The
	specification must clearly define what scenarios are in scope
	and what are out of scope in order to be interpreted,
	implemented, and tested correctly."

Replace "clearly define" with "define" to make this checkpoint testable.
In general, avoid adjectives in requirements as they have no value and
may do harm.

Search the guidelines for the word "clear" to find other places that
need improvement (to start with).


	"Checkpoint 1.2. Include User Scenarios. [Priority 2]"
	"Checkpoint 1.3. Provide examples. [Priority 1]"

These should have lower priority (priority 3?), IMO. Let specs
concentrate on defining the standard. Let text books, articles, and
such provide illustrations and examples. Ideal specs have
non-normative statements only to make the text readable/understandable
by smart humans, IMO.

	"The more numerous the examples, the better it is for the
	specifications comprehensibility."

I strongly disagree. The more non-normative scenarios/examples there
are, the higher the chance of a conflict between a normative statement
and a non-normative example. Regardless of what the spec says about such
conflicts and their resolution, they do lead to implementation bugs
because some developers will look at the example and ignore or
misinterpret the normative language.

Ideally, there should be a single [normative] definition for
everything in scope. No less, no more.


	"Checkpoint 1.4. Ensure that every test assertion is covered by
	an example. [Priority 3]"

Why?! Satisfying this requirement will make protocol and similar specs
huge without any added benefit. If, for example, a BNF specifies the
protocol header syntax, why would anybody want to generate an example
for everything that BNF covers?!

	"Checkpoint 1.5. Include an interpretation section. [Priority 2]
	It is hard to understand the formal descriptions of content
	without informative interpretations. The recent complex
	specifications like XML Schema [XML-SCHEMA] and XML Protocol
	have shown an necessity to have a "Primer" part or section to
	illustrate how to use the specification."

This should have lower priority (priority 3?). If somebody finds it
difficult to understand the formal descriptions, they should not rely on
the specs to help them. See above for reasons not to mix normative and
non-normative language and to avoid non-normative language. A short
informal "Introduction" section per spec is fine. An informal
interpretation for each formal description is very bad.

         "G 2. Identify what needs to conform and how.
         G 3. Address the use of profiles to divide the technology.
         G 4. Address the use of modules to divide the technology."

I would shorten the three guidelines above into a single guideline with
3-5 paragraph of text addressing a single (and the only, IMO) important
issue: "G 2. Define conformance".  That is, the spec MUST define what it
means to conform to the spec. Given a device/document/software within
the scope of the spec, it SHOULD be possible to test for conformance.
That's all.

There is no use for SpecGL to enumerate possible products that might
be in scope of future specs. There is no use to talk at length about
profiles and modules, their benefits and side-effects. Leave these
arguments for the spec authors. You are not going to provide a clear
one-size-fits-all answer any way. Demand only what is essential and
universal: a conformance clause.

In other words, the guidelines should mandate that the spec:
	- MUST define what is in scope
	- for things that are in scope,
	  MUST define what it means to conform
	- SHOULD make it possible to test for conformance
	  (note: not "to verify conformance")


	"Checkpoint 5.1. Make it clear ..."
	"Checkpoint 5.2. Make it clear ..."
	"Checkpoint 5.3. Make it clear ..."


See about the use of adjectives in checkpoints.


	"Guideline 6. Clarify the relation between deprecated features
	and conformance."

This guideline talks about one specific problem that some specs might
encounter. I suggest that this guideline is removed because the other
guidelines and checkpoints already cover what this guideline,
essentially, talks about: the spec must define conformance. A good
definition will cover deprecated features as needed, of course.


	"Guideline 7. Address the use of functional levels to divide the
	technology."

Again, I see no practical use from inclusion of this guideline. If the
specs have to use levels, they will. If they do not have to, they will
not. This is not general enough for this guidelines document, IMO.


	"Checkpoint 8.4. Include a statement regarding consistent
	handling of a discretionary item within an implementation.
	[Priority 2]

        The effect of each individual discretionary item should be
	consistent within a single implementation. For example, a
	browser's rendering of a XSL-FO (XSL Formatting Object) should
	be the same for every invocation regardless of the document
	instance."

Why on Earth would these guidelines require something specific like
that? Why handling of a discretionary item within an implementation
should be consistent? Care to define an implementation in this context?
Is browser X that is user-configured to do FOO the same implementation
as browser X that is user-configured to do BAR? Care to define an
invocation in this context? Can browser display cached documents
differently from first-hand responses? Etc., etc. This is just too
low-level to provide a general guideline, IMO.

	"Guideline 9. Allow extensions or NOT!"

This guideline sounds cool, but should be removed because it does not
cover any new ground, IMO: If the spec explicitly mentions an extension,
it MUST document compliant behavior just like for any other
non-extension, due to other guidelines. The problem arises only when the
spec implicitly allows something without defining a conformant reaction
to that something. That problem should be addressed by more general
guidelines:

	- Define compliant behavior for every conforming
	  object that the spec is using
	  (no "implicity", good "input")

	- Define compliant behavior for every non-conforming
	  object that an implementation may encounter
	  (no "implicity", bad "input")



	"Checkpoint 9.5. If extensions are allowed, register or publish
	them. [Priority 3]"

In general, this is totally unnecessary and introduces too much overhead
provided the good/bad input guidelines above are satisfied.


	"Checkpoint 10.5. Identify all dimensions of variability that
	are not used. [Priority 1]

	It must be possible for the reader to determine, from the
	conformance clause, if:

             there is only one class of product specified,
             there are no profiles,
             there is no division of the technology into modules,
             there are no deprecated features,
             there are no functional levels,
             there are no discretionary items,
             extensions are not allowed. "


I think this is an unnecessary overkill. The scope and conformance
sections MUST define scope and conformance. We should not require
authors to define non-scope and non-conformance as well. Everything that
is not covered by the scope, is out of scope. Everything that does not
conform, does not. What good does it make to state *in the conformance
clause* that, say, ``extensions are [not] allowed''?! The reader still
has to read the specs to understand what an extension is anyway. While
doing so, the reader will encounter specific conformance requirements
for those extensions, if any.

	"Checkpoint 11.2. Provide specific wording of the claim.
	[Priority 3]

        A well-formed conformance claim includes: date of the claim,
	specification name, date and version, URI of the specification,
	conformance level satisfied, and information about the subject
	(that which is claiming conformance). Information about the
	subject refers to information such as, the name of the software
	or software component, version information, and operating
	environment."

This is good but probably impractical. Even W3C "valid *ML" images do
not (and cannot) satisfy this verbose format.


	"Guideline 12. Publish an Implementation Conformance Statement
	proforma."

This is useful in some cases, but has nothing to do with the spec
itself, IMO. I would remove this guideline.

Also, keep in mind that it is impractical do provide a 1:1 mapping
between an implementation and conformance claim. A vendor is not going
to update their claim every time they do a minor software release, for
example. They would just state that "product X complies with FOO"
(see Checkpoint 12.2).

	"Checkpoint 13.2. Distinguish normative and informative text.
	[Priority 2]"

This should have the highest priority (Priority 1?), IMO. The spec is no
good if normative and informative statements are indistinguishable.

	"Finally, using an advanced schema for specification authoring
	allows for greater control over coverage of the specification in
	the Test Suite."

Not really. It may make greater control over coverage easier to
implement, that's all. Plain ASCII text allows for the same coverage
control as *ML.


	"Guideline 15. Include test assertions."

IMO, the specs should not include test assertions because there should be
a single normative definition and there should be nothing else. This is
similar to including informative examples and such. The rationale the
guidelines give for including Guideline 15 are not really specific to
the inclusion of test assertions into the spec:

	"Including test assertions as part of the specification
	facilitates and promotes the development of test materials."

So lets include everything that facilitates and promotes then. Let's
include translations to other languages, alternative formats, author
commentary, code samples, etc. etc.

	"Test assertions (that are part of a specification) are publicly
	 reviewed and agreed to"

So can be test assertions published outside of the spec.

	"Detailed control over what parts of a specification an
	 implementation supports, grouped by behavior, technical
	 aspect (method/interface) or other conceptual grouping;"

This should be provided by normative definitions, not test assertions.

	"Detailed reporting when testing an implementation using the
	test;"

Unrelated to the inclusion of test assertions into the spec.

Moreover, I can make an argument that including test assertions in the
spec may cripple future test suites and even implementations!  Spec
implementors will see the assertions and make sure they write code that
passes those test cases. Test suite developers will implement the same
assertions that the implementors were looking at (at least they will
start with them, many will not go beyond). As a result, the
effectiveness of a test suite will be minimal. It is a well known rule
of thumb that best testing is done by non-implementors. Including
"standard" test assertions violates this rule in practice (in theory it
would not matter, of course).


	"The test assertions of this Specification Guidelines document
	are found in the prioritized checkpoints. A checkpoint will
	contain at least one, and may contain multiple individual
	requirements. These requirements are the test assertions of this
	specification."

Note that most of what you call "test assertions" cannot be tested with
because they contain undefined adjectives like "clear".


	"test assertion: any sentence or equivalent assemblage of
	 words and phrases that prescribes the behavior that must be
	 obtained when a stimulus occurs under a certain set of
	 conditions. (See also QA Glossary [QA-GLOSSARY].)"

Can a test assertion be expressed in a formal language without words
and phrases rather than in sentences of a human language?


Thank you,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Friday, 30 August 2002 17:06:55 UTC