Re: Review of QA Spec guidelines

Review of spec guidelines.  Note that Ms. Lewis is known for her occasional tendency toward shrewishness, especially in critical reviews.  In other words, this might need some serious editing (or wholesale revision) before release.

General comments:

1. All specs include seven "levels" of conformance.  However, only three levels are actually public; these are called "priorities".  Collapse the seven levels into the three real ones (now called priorities), since these are the basis of conformance measurement.

2. Guidelines two through nine are grouped as "dimensions of variability," and referred to as such by themselves and by other guidelines.  If the concept of dimensions of variability is of this much importance, it should be reflected in the structure of the document: guidelines two through nine should be grouped, structurally, as "dimensions of variability".

3. In general, the language is insufficiently rigorous/precise for a common use case, and the model may be explained in text (see 2, above) rather than structurally.  The problem is that, while many WG members may read the entire spec, others may be tasked with judging conformance to a particular guideline, or even a particular checkpoint.  Insofar as is possible, then, guidelines and checkpoints should contain sufficient definition for local understanding, and pointers to all related items in the document.  That is, a reader should be able to enter via any checkpoint, and gather all and only the information needed to understand the checkpoint starting from that entry point.

4. Spell and grammar check.  There are a number of problems with both, which probably do not deserve direct mention.  Also, the style of the prose occasionally changes radically (from a formal, romance-language-influenced european english to a very colloquial american style).  This is jarring.

5. If abbreviated forms are used to refer to other documents, these abbreviations ought to also appear in the bibliography.  In general, reference to a document should always be a <bibloc>.

Specific comments, by section number:

1.general: terseness in specifications is *not* of minimal importance.

1.3 Bullet lists are there to call out important items.  An eight-item list should either be shortened or reformatted; it fails as bullet points.

1.6 Why isn't the glossary here, instead of at the end?

1.7 Checkpoint priorities are "levels", in the sense that guideline 6 defines levels.

1.8 Drop this, and restructure.  If it's important enough to occupy a full page here, and be referred to without further explanation elsewhere, it's important enough to restructure so as to identify clearly the membership of DoV.  Wherever dimensions of variability are mentioned in the document, it should be possible to hyperlink to the appropriate place (which means a briefer discussion, as an introduction to the collection of guidelines 2-9).

2.general: guidelines escape the section/technical numbering of the remainder of the document.  Why isn't guideline 1 section 2.1?  And checkpoint 1.2 section 2.1.2?  This is going to cause confusion, when others make reference to the document: if I say 3.2, do I mean a checkpoint or a section?  Section 2 is by far the bulk of the document; perhaps each guideline ought to be given a section number instead (guidelines 2 through 15, then, instead of 1 through 14).

G1: what's the difference between examples and illustrations?  (Checkpoint 1.2 versus checkpoint 1.4)

G2: The use of categories versus classes of products is altogether unclear.  If categories and classes of products are to be called out, normatively, then these should have status in the TOC (the list of categories and the list of classes should both have ids, be targets for hyperlinks, and should have subheadings to identify them visually).

G2: The checkpoints based on classes of product and categories are awkward, because the use of the existing enumeration is REQUIRED but only if applicable.  That is, the use of the existing enumerations is COMPLETELY OPTIONAL.  The language should reflect that it leaves an enormous loophole for spec authors to ignore the existing lists.

G3.general: why should conformance policy be considered a dimension of variability?  It is potentially partitioned by those dimensions, but does not introduce its own dimensions.

G3: the priorities here are very strange.  Why is justifying the use of a dimension priority one, while establishing a minimum requirement is priority two?

CP4.2: again, minimal conformance requirements are only priority two.

G5: Unlike G4, which notes that profiles may be a point of extension, G5 does not consider modules to be a point of extension.  In the web services world, "modules" certainly are a point of extension, and so have rules for defining new modules (just as, in G4, there are assertions associated with rules for defining new profiles).  The document should recognize this.

G6: Levels are the primary defining mechanism for the QA framework, but there is only one checkpoint here.  There should also be (at least) a checkpoint to establish that levels create a hierarchy of conformance; that the more advanced levels include the earlier levels (thus establishing that there is a minimum level).

CP7.3: awkward.  This checkpoint basically demands that if some feature has been deprecated, possibly because no one can agree on its proper definition, then it must be properly defined and its interactions fully characterized.  It is likely that some things will be deprecated precisely because they cannot be well and unambiguously characterized; this checkpoint ensures that as long as these features are part of the main spec, it can conform at priority two, but as soon as the features are deprecated, it cannot.

G9: The position of the QA framework WG, that extensions should not be allowed, is quite clear.  This is a political position, and it is irritating to those working on specifications that clearly demand public extensibility.  Discourage elsewhere.  In the guidelines, describe conformance.  This is particularly irritating in that these guidelines are, themselves, extensible.

CP9.6: Unnaceptable.  Extensions may be allowed in order to permit new functionality to be introduced and tested prior to standardization.  There may not be any alternatives (interoperable or otherwise) to the use of a particular extension, and in particular, it is completely impossible for any specification that permits extensions to supply a workaround to the use of every uninvented extension imaginable.  In other words, no specification that allows extensions can conform at priority three, ever.

G10: This is G3.  Or it should be.  Or G3 belongs in Ops, and G10 here.  If a policy has been established, then it has been documented as well; the two are inextricable, when considering a written specification.

CP10.1: Unacceptable.  The QA specification does not conform to this checkpoint, which is priority one.  It must be acceptable to place the conformance requirements in each section of the document.  For instance, it must be acceptable to place the conformance requirements for each checkpoint in a QA document inside the checkpoint.

G11: This is also still part of G3/G10.  This section should also discuss how one describes "module conformance" versus "profile conformance" versus "level conformance" versus "conforming extension", or how these might be combined.  Perhaps this information is in the examples (I haven't looked).

G13: The division of the various conformance-related guidelines into three, ten, eleven, and thirteen is not entirely clear, nor is it entirely clear that these separate sections do not contradict one another.

3.1 "All sentences using a capitalized keyword from RFC 2119" ... this is a totally strange definition of how to identify normative text.  Since the definitions of terms are not normative, I am free to redefine them however I wish, and claim whatever conformance I care to (courtesy Orwell's Elder Sibling Software Creation Consortium).  Note that the stated priorities for each checkpoint are not normative.  This definition, in other words, is just awful.

3.4 This is completely absurd and pointless, as pointless as the creation of four levels that will never be used.  A-conforming = priority one = level three; AA-conforming = priority two = level five; AAA-conforming = priority three = level seven.  Pick one set of terms and toss the remainder.  (N.B.: the association of priorities with levels is a consequence of Ops guideline 1).

4. To avoid the ridiculous problems introduced in section 3.1, change it to include, as normative, all of sections 1 and 4, which define terms.  At least.  Or create a better definition.

-- 
Amelia A. Lewis
Architect, TIBCO/Extensibility, Inc.
alewis@tibco.com

Received on Friday, 14 March 2003 11:49:43 UTC