QA Framework: Specification Guidelines

W3C Working Draft 10 February 2003

Review-assignment skeleton for document version:
http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/

Reviewed by: Mark Skall

Guidelines and Checkpoints

Guideline 1. Define Scope.

1.1 Include the scope of the specification. [Priority 1]

Conformance requirements: the specification MUST define the subject matter of the specification, and SHOULD include a statement of objectives and/or goals. This information MUST appear at the beginning of the document

Conforms.

This is a big improvement over the last time I assessed conformance. The wording is no longer vague. One suggestion – Use the terms “normative” and “informative” (or “non-normative”) when describing the checkpoints, ExTech and the appendices. This makes it clearer what is and isn’t required.

1.2 Illustrate what is in scope [Priority 2]

Conformance requirements: the specification MUST include examples or use cases illustrating what is in or out of scope.

Conforms.

ExTech provides examples and use cases.

1.3 Provide Usage Scenarios. [Priority 3]

Conformance requirements: the specification MUST provide Usage Scenarios

Conforms.

ExTech provides examples and use cases. Question – What is the difference between Checkpoint 1. 2 and 1.3 (i.e., what’s the difference between use cases and usage scenarios?) Do we need both checkpoints?

1.4 Provide examples. [Priority 2]

Conformance requirements: the specification MUST provide one or more examples of the functionality, concepts, and behaviors of the specification.

Conforms.

ExTech and narrative descriptions provide this functionality.

Guideline 2. Identify what needs to conform and how.

2.1 Identify all classes of product. [Priority 1]

Conformance requirements: a specification:

Conforms.

Again, there is much improvement over the last time this requirement was reviewed. The scope is much more precise and has a section addressing classes of products. “Technical Reports”, the class of product for SpecGL should probably be added to the enumerated list of classes of products described later in Guideline 2.

However, I’m not too sure about adherence to the second and third “MUST” above (especially the third). Although the specific categories don’t apply to categorizing SpecGL, I believe, according to our requirements, there should be some reference to these categories in Section 1.2 and an explanation of SpecGL’s classes of products vis a vis the enumerated list (i.e, explicitly state that this is a new class of product). (Again, this could also be achieved by adding “Technical Reports” to the enumerated list in Guideline 2.)

2.2 For each class of product, define the conformance requirements. [Priority 1]

Conformance requirements: the specification MUST define the conformance requirements for each identified class of product.

Conforms.

The discussion on degrees of conformance (A, AA, AAA) in the conformance section clearly fulfill this checkpoint.

2.3 Identify which of the categories of object are specified in the document as a whole. [Priority 3]

Conformance requirements: the specification MUST identify in the Introductory section which of the categories of object are specified in the document as a whole.

Does not conform (but can be made to conform easily).

“Categories of objects” is still a poorly defined term. They should be called “category of specifications”. Although SpecGL does not fall within any of the categories enumerated, I believe this requirement says that SpecGL should explicitly state what “category” it is (Framework Guideline?).

2.4 If there are several classes of products, define their relationships and interaction with other dimensions of variability. [Priority 2]

Conformance requirements: the specification MUST address and discuss the relations and interactions between classes of product and the other dimensions of variability. It is not applicable if there is only one class of product.

N/A – Only one class of product.

Guideline 3. Specify conformance policy.

3.1 Specify any universal requirements for minimum functionality. [Priority 2]

Conformance requirements: the specification MUST include a normative section enumerating the minimal requirements that apply across all identified products of a class.

Conforms.

One can deduce from SpecGL’s conformance section that conforming to all priority 1 checkpoints are the minimal requirements. However, it would be clearer if SpecGL said exactly that.

3.2 If special conformance terms are used, include a definition in the specification. [Priority 1]

Conformance requirements: the specification MUST be defined, either by reference or by including the definition in the text.

Conforms. Terms like DOV and ICS are defined.

3.3 Justify any usage of a dimension of variability [Priority 1]

Conformance requirements: a specification MUST justify each Dimension of Variability the specification uses.

Conforms. Extensibility is the only DOV used. They are justified by the rationale in Section 3.2.

One can argue that A, AA, AAA are levels, but I don’t think they fit the intent of what we meant be DOV levels.

Guideline 4. Address the use of profiles to divide the technology.

Since SpecGL does not use profiles, this guideline does not apply.

4.1 Indicate whether or not the use of profiles is mandatory for conformance. [Priority 1]

N/A.

4.2 If profiles are chosen, define any minimal requirements. [Priority 2]

N/A.

4.3 If profiles are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]

N/A.

4.4 If profiles are chosen, address rules for profiles. [Priority 2]

N/A.

Guideline 5. Address the use of modules to divide the technology.

Since SpecGL does not use modules, this guideline does not apply.

5.1 If modules are chosen, indicate any mandatory conditions or constraints on their usage. [Priority 1]

N/A.

5.2 If modules are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]

N/A.

Guideline 6. Address the use of functional levels to divide the technology.

Since SpecGL does not use functional levels, this guideline does not apply.

6.1 If levels are used, define their relationships and interaction with other dimensions of variability. [Priority 2]

N/A.

Again, if we consider the degrees of conformance to be levels, SpecGL would conform since our section on Extensibility discusses the relationship to degrees of conformance A, AA, and AAA.

Guideline 7. Identify the relation between deprecated features and conformance.

Since SpecGL does not use deprecated features, this guideline does not apply.

7.1 Identify each deprecated feature. [Priority 1]

N/A.

7.2 For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation. [Priority 1]

N/A.

7.3 If deprecation is used, define its relationships and interaction with other dimensions of variability. [Priority 2]

N/A.

7.4 Include an explanation for the deprecation. [Priority 3]

N/A.

7.5 Include examples to illustrate how to avoid using deprecated features. [Priority 3]

N/A.

Guideline 8. Define discretionary items.

In many sections, especially here, it’s difficult to map rec type terms, like discretionary items, to SpecGL. However, it seems like many of the DOV are discretionary items. If so, SpecGL does not conform to these requirements.

8.1 State the circumstances for when discretionary items are allowed [Priority 1]

Conformance requirements: the specification MUST indicate the rationale for the discretionary items and MUST identify by labeling all discretionary items. It is not applicable for specifications that do not have discretionary items.

Doesn’t Conform.

Profiles, modules, levels, etc. are optional features and, thus, discretionary items. Although some rationale for inclusion can be inferred for profiles and levels, there is no rationale for the inclusion of modules. Also, these are not labeled, in SpecGL, as discretionary items, as they are required to be.

8.2 For implementation dependencies, address the allowable differences between implementations [Priority 1]

N/A.

8.3 Indicate any constraints on the number of choices/options that can be implemented for discretionary choices [Priority 2]

N/A.

8.4 Promote consistent handling of discretionary choices. [Priority 2]

N/A.

8.5 If discretionary items are used, define their relationships and interaction with other dimensions of variability. [Priority 2]

Conformance requirements: the specification MUST address and discuss the relations and interactions between profiles and all the other DoV.

N/A, although the checkpoint doesn’t match the conformance requirements.

The checkpoint asks to define relationships among DOVs if discretionary items are used (which they are in SpecGL). However, the conformance requirements only address the relationships between profiles and DOV.

Guideline 9. Allow extensions or NOT!

9.1 Indicate if the specification is extensible. [Priority 1]

Conformance requirements: the specification MUST state the conditions under which extensions are allowed and disallowed.

Conforms.

Section 3.2 does an excellent job of explaining the extensibility of SpecGL and the specific requirements.

9.2 If extensions are allowed, define their scope and constraints. [Priority 1]

Conformance requirements: the specification MUST state

Conforms.

Again, Section 3.2 covers these items. However, it’s not clear what is meant by the must requirement for “scope of the extensions” and “any limitations or restrictions on the use of the extension”. I don’t think SpecGL addresses these requirements; however, it can be inferred that the “scope” is unlimited and there are no “limitations or restrictions”.

9.3 Prevent extensions from contradicting the specification. [Priority 1]

Conformance requirements: the specification MUST state that extensions can not negate or change support for required functionality. It is not applicable if extensions are not allowed.

Conforms.

Section 3.2 contains an explicit statement addressing this, as required.

9.4 Define a uniform mechanism to create an extension [Priority 3]

Conformance requirements: the specification MUST provide a uniform way to define that extensibility is being invoked and MUST provide the syntax to be used to indicate the extension. It is not applicable if extensions are not allowed.

Doesn’t conform.

SpecGL does not define a standard way to define extensions.

9.5 Require that extensions be published. [Priority 3]

Conformance requirements: the specification MUST require that the syntax and semantics of the extension be publicly documented. It is not applicable if extensions are not allowed.

Doesn’t conform.

SpecGL does not require extensions to it be published.

9.6 Require that implementations provide interoperable alternatives to extensions [Priority 3]

Conformance requirements: the specification MUST indicate via conformance requirements that implementations provide a mode under which they produce only conforming content. This checkpoint is not applicable if extensions are not allowed.

Doesn’t conform.

SpecGL does not provide a mode to produce only conforming content.

9.7 If extensions are allowed, define their relationships and interaction with other dimensions of variability [Priority 2]

Conformance requirements: the specification MUST address and discuss the relations and interactions between extensions and all the other DoV.

N/A since there are no other DOVs besides extensions.

Guideline 10. Provide a conformance clause.

10.1 Include a conformance section. [Priority 1]

Conformance requirements: the specification MUST document its conformance policy and specific conformance requirements in a dedicated section of the document.

Conforms.

Section 3 (Conformance) is a very comprehensive conformance section.

10.2 Make normative reference to specifications on which the current specification depends [Priority 2]

Conformance requirements: the specification MUST have normative references to any identified specification it depends on and MUST describe the relationship between the specifications and any identified conformance implications

Conforms?

SpecGL makes a normative reference to RFC2119, upon which it depends for key word definitions and references RFC2119 in the conformance section. Also, the references to ExTech have been rewritten to explicitly state that ExTech is informative, so that is no longer an issue. However, Section 1.4 is a little fuzzy. This section says that the QA framework documents are interrelated (thus they depend on each other?). This Section should explictly state that there are no conformance inmplications with respect to the other documents.

Guideline 11. Specify how to make conformance claims.

Again, with minor changes, SpecGL can conform to this Guideline.

11.1 Identify and define all conformance designations. [Priority 1]

Conformance requirements: the specification MUST identify and define all the conformance designations used.

Conforms.

This is clearly done in Section 3.4 with the A, AA, AAA degrees of conformance.

11.2 Provide specific wording of the claim. [Priority 3]

Conformance requirements: the specification MUST provide specific wording of the claim and the specific wording MUST at least include the specification name, its version, the conformance level satisfied and information about the subject that which is claiming conformance and the date of the claim.

Doesn’t conform.

SpecGL attempts to meet this requirement in Section 3.4 with the wording “To make an assertion about conformance to this document, specify: . . ” SpecGL is missing the version and information about the subject that which is claiming conformance.

11.3 Provide a conformance disclaimer. [Priority 3]

Conformance requirements: the specification MUST provide a conformance disclaimer.

Conforms.

This is clearly fulfilled in Section 3.5.

11.4 Impose no restrictions about who can make a claim or where claims can be published. [Priority 1]

Conformance requirements: the specification MUST NOT include any restriction about who can make a claim nor where claims can be published.

Conforms.

No restrictions are present.

Guideline 12. Publish an Implementation Conformance Statement proforma.

12.1 Provide an Implementation Conformance Statement proforma. [Priority 3]

Conformance requirements: the specification MUST either:

Conforms.

The list of checkpoints is the ICS. This is explicitly stated in Section 3.4

12.2 Require the ICS be completed as part of the conformance claim. [Priority 3]

Conformance requirements: the specification MUST include the ICS as part of its conformance claim requirements. This checkpoint is not applicable, if an ICS is not applicable to the specification.

Conforms.

This requirement is also met in Section 3.4

Guideline 13. Clearly identify conformance requirements.

13.1 Use conformance key words. [Priority 1]

Conformance requirements: the specification MUST use RFC 2119 keywords to denote whether or not requirements are mandatory, optional, or suggested.

Conforms.

Done throughout by using MUST, SHOULD.

13.2 Distinguish normative and informative text. [Priority 1]

Conformance requirements: the specification MUST distinguish normative text from informative text.

Conforms.

This is a big improvement from the last version I reviewed for conformance.

13.3 Use consistent terminology. [Priority 1]

Conformance requirements: the specification MUST use identical wording to express identical provisions, and analogous wording to express analogous provisions .

Conforms, I guess.

However, it’s not clear to me exactly what this checkpoint is requiring. What is meant, for instance, by “analogous wording to express analogous provisions” and how can one test for this? This checkpoint is pretty much unverifiable and untestable.

13.4 Provide a fast way to find conformance information [Priority 2]

Conformance requirements: the specification MUST provide at least one navigation mechanism that allows the reader to locate by direct access, all conformance-related information that is relevant to the specification. The mechanism MUST minimally locate:

Conforms.

The Table of Contents satisfies this requirement.

Guideline 14. Provide test assertions.

Despite the claim that test assertions are present, I don’t believe that the prioritized checkpoints fulfill the definition of test assertions. In fact, the checkpoints are really the requirements, analogous to functional requirements in recs.

14.1 Provide test assertions [Priority 2]

Conformance requirements: the specification MUST provide a normative list of test assertions stated in it.

Doesn’t Conform.

Although SpecGL specifically asserts that the test assertions are found in the prioritized checkpoints, I don’t believe this meets the definition of “test assertion”, as defined and explained in SpecGL. SpecGL defines test assertion as “A test assertion is a statement of behavior, action or condition that can be measured or tested. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification.”

The problem with using the checkpoints as fulfilling the test assertion requirement are many. I’ll mention two: 1) Test assertions are supposed to be “derived” from the spec’s requirements. The checkponts are the spec’s requirements. 2) Some of these checkpoints can not be measured or tested (see checkpoint 13.3 (analagous) as one example).

In addition, I do not believed that these checkpoints are specific enough to be a “complete, testable statement”. See my previous write-up on Test Assertions for guideline 3 at http://lists.w3.org/Archives/Public/www-qa-wg/2002Dec/0104.html.

14.2 Provide a mapping between the specification and the test assertions list [Priority 2]

Conformance requirements: the specification MUST provide a mechanism linking each test assertion to the part of the specification it is stated.

N/A, since I don’t believe test assertions are provided.

No mapping or mechanism is provided.

Conclusion

This version is a big improvement over the previous version. However, there are still some serious problems.

  1. Guideline 8 is not met since SpecGL’s discretionary items (profiles, modules, etc.) are not labeled as such and some do not include the required rationale.
  2. Guideline 9 (extensibility) has some problems. There is no uniform mechanism to create extensions and SpecGL does not require that extensions be published.
  3. Guideline 11 is not met because specific wording of the conformance claim has some missing elements.
  4. Guideline 14 (test assertions) doesn’t conform

These are the areas that I said needed to be addressed during my last review, with an annotation of whether or not it has now been addressed:

  1. Rewrite of the narrative to eliminate ambiguity and delineate requirements;

    a. This has been done.

  2. Categories and classes of products enumerated need to be updated to include “specifications” (or “Framework Guideline”) as a category and a product.

    a. This has not been done.

  3. An explicit statement saying “strict conformance in not required” needs to be added;

    a. No longer necessary due to re-write of SpecGL.

  4. There should be a discussion of the circumstances (rationale for inclusion) under when discretionary items (DOV) are allowed;

    a. This has not been done.

  5. SpecGL needs to discuss extensions and state that they are allowed and then satisfy the other checkpoints in Guideline 9

    a. For the most part this has been done. However, the checkpoints addressing providing a uniform way to handle extensions and publishing them not been met.

  6. There needs to be a statement as to why an ICS is not needed or (preferably) an ICS should be developed and included.

    a. This has been done.

  7. Some requirements need to be rewritten to be more specific and testable (e.g., “analogous wording to express analogous provisions”)

    a. This has not been done.

  8. Test assertions need to be added (or SpecGL needs to say assertions are not applicable to Guideline documents).

    a. This has not been done.