- From: Mark Skall <mark.skall@nist.gov>
- Date: Fri, 04 Apr 2003 09:44:57 -0500
- To: www-qa-wg@w3.org
- Message-Id: <5.1.0.14.2.20030404092618.01e5ba40@mailserver.nist.gov>
To all,
Following is my assessment of SpecGL's present degree of conformance to
itself. At the end of the assessment I've included conclusions and an
update of what has been fixed since my last assessment. In summary,
SpecGL's conformance is decidedly better than when I last reviewed it, but
there are still some key areas where SpecGL does not conform.
Mark
Assessment of Conformance of SpecGL to SpecGL
QA Framework: Specification Guidelines W3C Working Draft 10 February 2003
Guidelines and Checkpoints
Guideline 1. Define Scope.
1.1 Checkpoint 1.1. Include the scope of the specification. [Priority 1]
Conformance requirements: the specification MUST define the subject matter
of the specification, and SHOULD include a statement of objectives and/or
goals. This information MUST appear at the beginning of the document
Conforms.
This is a big improvement over the last time I assessed conformance. The
wording is no longer vague. One suggestion Use the terms “normative” and
“informative” (or “non-normative”) when describing the checkpoints, ExTech
and the appendices. This makes it clearer what is and isn’t required.
1.2 Checkpoint 1.2. Illustrate what is in scope. [Priority 2]
Conformance requirements: the specification MUST include examples or use
cases illustrating what is in or out of scope.
Conforms.
ExTech provides examples and use cases.
1.3 Checkpoint 1.3. Provide Usage Scenarios. [Priority 3]
Conformance requirements: the specification MUST provide Usage Scenarios
Conforms.
ExTech provides examples and use cases.
Question What is the difference between Checkpoint 1. 2 and 1.3 (i.e.,
what’s the difference between use cases and usage scenarios?) Do we need
both checkpoints?
1.4 Checkpoint 1.4. Provide examples. [Priority 2]
Conformance requirements: the specification MUST provide one or more
examples of the functionality, concepts, and behaviors of the specification.
Conforms.
ExTech and narrative descriptions provide this functionality.
Guideline 2. Identify what needs to conform and how.
2.1 Checkpoint 2.1. Identify all classes of product. [Priority 1]
Conformance requirements: a specification:
· MUST list the classes of product it addresses
· MUST use the above enumerated classes names when they
match those of the specification
· MUST define and describe identified product classes that
does not match the enumerated set
Conforms.
Again, there is much improvement over the last time this requirement was
reviewed. The scope is much more precise and has a section addressing
classes of products. “Technical Reports”, the class of product for SpecGL
should probably be added to the enumerated list of classes of products
described later in Guideline 2.
However, I’m not too sure about adherence to the second and third “MUST”
above (especially the third). Although the specific categories don’t apply
to categorizing SpecGL, I believe, according to our requirements, there
should be some reference to these categories in Section 1.2 and an
explanation of SpecGL’s classes of products vis a vis the enumerated list
(i.e, explicitly state that this is a new class of product). (Again, this
could also be achieved by adding “Technical Reports” to the enumerated list
in Guideline 2.)
2.2 Checkpoint 2.2. For each class of product, define the conformance
requirements. [Priority 1]
Conformance requirements: the specification MUST define the conformance
requirements for each identified class of product.
Conforms.
The discussion on degrees of conformance (A, AA, AAA) in the conformance
section clearly fulfill this checkpoint.
2.3 Checkpoint 2.3. Identify which of the categories of object are
specified in
the document as a whole. [Priority 3]
Conformance requirements: the specification MUST identify in the
Introductory section which of the categories of object are specified in the
document as a whole.
Does not conform (but can be made to conform easily).
“Categories of objects” is still a poorly defined term. They should be
called “category of specifications”. Although SpecGL does not fall within
any of the categories enumerated, I believe this requirement says that
SpecGL should explicitly state what “category” it is (Framework Guideline?).
2.4 Checkpoint 2.4. If there are several classes of products, define their
relationships and interaction with other dimensions of variability. [Priority2]
Conformance requirements: the specification MUST address and discuss the
relations and interactions between classes of product and the other
dimensions of variability. It is not applicable if there is only one class
of product.
N/A Only one class of product.
Guideline 3. Specify conformance policy.
3.1 Checkpoint 3.1. Specify any universal requirements for minimum
functionality. [Priority 2]
Conformance requirements: the specification MUST include a normative
section enumerating the minimal requirements that apply across all
identified products of a class.
Conforms.
One can deduce from SpecGL’s conformance section that conforming to all
priority 1 checkpoints are the minimal requirements. However, it would be
clearer if SpecGL said exactly that.
3.2 Checkpoint 3.2. If special conformance terms are used, include a
definition in the specification. [Priority1]
Conformance requirements: the specification MUST be defined, either by
reference or by including the definition in the text.
Conforms. Terms like DOV and ICS are defined.
3.3 Justify any usage of a dimension of variability [Priority 1]
Conformance requirements: a specification MUST justify each Dimension of
Variability the specification uses.
Conforms. Extensibility is the only DOV used. They are justified by the
rationale in Section 3.2.
One can argue that A, AA, AAA are levels, but I don’t think they fit the
intent of what we meant be DOV levels.
Guideline 4. Address the use of profiles to divide the technology.
Since SpecGL does not use profiles, this guideline does not apply.
4.1 Checkpoint 4.1. Indicate whether or not the use of profiles is mandatory
for conformance. [Priority 1]
N/A.
4.2 Checkpoint 4.2. If profiles are chosen, define any minimal requirements.
[Priority 2]
N/A.
4.3 Checkpoint 4.3. If profiles are chosen, define their relationships and
interaction with other dimensions of variability. [Priority 2]
N/A.
4.4 Checkpoint 4.4. If profiles are chosen, address rules for profiles.
[Priority 2]
N/A.
Guideline 5. Address the use of modules to divide the technology.
Since SpecGL does not use modules, this guideline does not apply.
5.1 Checkpoint 5.1. If modules are chosen, indicate any mandatory
conditions or
constraints on their usage. [Priority 1]
N/A.
5.2 Checkpoint 5.2. If modules are chosen, define their relationships and
interaction with other dimensions of variability. [Priority 2]
N/A.
Guideline 6. Address the use of functional levels to divide the technology.
Since SpecGL does not use functional levels, this guideline does not apply.
6.1 Checkpoint 6.1. If levels are used, define their relationships and
interaction with other dimensions of variability. [Priority 2]
N/A.
Again, if we consider the degrees of conformance to be levels, SpecGL would
conform since our section on Extensibility discusses the relationship to
degrees of conformance A, AA, and AAA.
Guideline 7. Identify the relation between deprecated features and conformance.
Since SpecGL does not use deprecated features, this guideline does not apply.
7.1 Checkpoint 7.1. Identify each deprecated feature. [Priority 1]
N/A.
7.2 Checkpoint 7.2. For each class of product, specify the degree of support
required for each deprecated feature and the conformance consequences of the
deprecation. [Priority 1]
N/A.
7.3 Checkpoint 7.3. Include an explanation for the deprecation. [Priority 3]
N/A.
7.4 Checkpoint 7.4. Include examples to illustrate how to avoid using
deprecated features. [Priority 3]
N/A.
Guideline 8. Define discretionary items.
In many sections, especially here, it’s difficult to map rec type terms,
like discretionary items, to SpecGL. However, it seems like many of the
DOV are discretionary items. If so, SpecGL does not conform to these
requirements.
8.1 Checkpoint 8.1. State the circumstances for when discretionary items are
allowed [Priority 1]
Conformance requirements: the specification MUST indicate the rationale for
the discretionary items and MUST identify by labeling all discretionary
items. It is not applicable for specifications that do not have
discretionary items.
Doesn’t Conform.
Profiles, modules, levels, etc. are optional features and, thus,
discretionary items. Although some rationale for inclusion can be inferred
for profiles and levels, there is no rationale for the inclusion of
modules. Also, these are not labeled, in SpecGL, as discretionary items,
as they are required to be.
8.2 Checkpoint 8.2. For implementation dependencies, address the allowable
differences between implementations [Priority 1]
N/A.
8.3 Checkpoint 8.3. Indicate any constraints on the number of choices/options
that can be implemented [Priority 2]
N/A.
8.4 Checkpoint 8.4. Promote consistent handling of discretionary choices.
[Priority 2]
N/A.
8.5 Checkpoint 8.5 If discretionary items are used, define their
relationships and interactions with other dimensions of variability.
Conformance requirements: the specification MUST address and discuss the
relations and interactions between profiles and all the other DoV.
N/A, although the checkpoint doesn’t match the conformance requirements.
The checkpoint asks to define relationships among DOVs if discretionary
items are used (which they are in SpecGL). However, the conformance
requirements only address the relationships between profiles and DOV.
Guideline 9. Allow extensions or NOT
9.1 Checkpoint 9.1. Indicate if the specification is extensible.
[Priority 1]
Conformance requirements: the specification MUST state the conditions under
which extensions are allowed and disallowed.
Conforms.
Section 3.2 does an excellent job of explaining the extensibility of SpecGL
and the specific requirements.
9.2 Checkpoint 9.2. If extensions are allowed, define their scope and
constraints. [Priority 1]
Conformance requirements: the specification MUST state
· the conditions under which extensions are allowed,
· the scope of the extensions,
· their effect on conformance claims,
· any limitations or restrictions on the use of the extension
and MUST document the rationale for allowing extensions by referencing use
cases and/or project requirements. It is not applicable if the
specification does not allow extensions.
Conforms.
Again, Section 3.2 covers these items. However, it’s not clear what is
meant by the must requirement for “scope of the extensions” and “any
limitations or restrictions on the use of the extension”. I don’t think
SpecGL addresses these requirements; however, it can be inferred that the
“scope” is unlimited and there are no “limitations or restrictions”.
9.3 Checkpoint 9.3. Prevent extensions from contradicting the specification.
[Priority 1]
Conformance requirements: the specification MUST state that extensions can
not negate or change support for required functionality. It is not
applicable if extensions are not allowed.
Conforms.
Section 3.2 contains an explicit statement addressing this, as required.
9.4 Checkpoint 9.4. Define a uniform mechanism to create an extension.
[Priority 3]
Conformance requirements: the specification MUST provide a uniform way to
define that extensibility is being invoked and MUST provide the syntax to
be used to indicate the extension. It is not applicable if extensions are
not allowed.
Doesn’t conform.
SpecGL does not define a standard way to define extensions.
9.5 Checkpoint 9.5. Require that extensions be published. [Priority 3]
Conformance requirements: the specification MUST require that the syntax
and semantics of the extension be publicly documented. It is not applicable
if extensions are not allowed.
Doesn’t conform.
SpecGL does not require extensions to it be published.
9.6 Checkpoint 9.6. Require that implementations provide interoperable
alternatives to extensions. [Priority 3]
Conformance requirements: the specification MUST indicate via conformance
requirements that implementations provide a mode under which they produce
only conforming content. This checkpoint is not applicable if extensions
are not allowed.
Doesn’t conform.
SpecGL does not provide a mode to produce only conforming content.
9.7 Checkpoint 9.7 If extensions are allowed, define their relationships
and interaction with other dimensions of variability.
Conformance requirements: the specification MUST address and discuss the
relations and interactions between extensions and all the other DoV.
N/A since there are no other DOVs besides extensions.
Guideline 10. Provide a conformance clause.
10.1 Checkpoint 10.1. Include a conformance section. [Priority 1]
Conformance requirements: the specification MUST document its conformance
policy and specific conformance requirements in a dedicated section of the
document.
Conforms.
Section 3 (Conformance) is a very comprehensive conformance section.
10.2 Checkpoint 10.2. Make normative reference to specifications on which the
current specification depends [Priority 2]
Conformance requirements: the specification MUST have normative references
to any identified specification it depends on and MUST describe the
relationship between the specifications and any identified conformance
implications
Conforms?
SpecGL makes a normative reference to RFC2119, upon which it depends for
key word definitions and references RFC2119 in the conformance
section. Also, the references to ExTech have been rewritten to explicitly
state that ExTech is informative, so that is no longer an issue. However,
Section 1.4 is a little fuzzy. This section says that the QA framework
documents are interrelated (thus they depend on each other?). This Section
should explictly state that there are no conformance inmplications with
respect to the other documents.
Guideline 11. Specify how to make conformance claims.
Again, with minor changes, SpecGL can conform to this Guideline.
11.1 Checkpoint 11.1. Identify and define all conformance designations.
[Priority 1]
Conformance requirements: the specification MUST identify and define all
the conformance designations used.
Conforms.
This is clearly done in Section 3.4 with the A, AA, AAA degrees of conformance.
11.2 Checkpoint 11.2. Provide specific wording of the claim. [Priority 3]
Conformance requirements: the specification MUST provide specific wording
of the claim and the specific wording MUST at least include the
specification name, its version, the conformance level satisfied and
information about the subject that which is claiming conformance and the
date of the claim.
Doesn’t conform.
SpecGL attempts to meet this requirement in Section 3.4 with the wording
“To make an assertion about conformance to this document, specify: . . ”
SpecGL is missing the version and information about the subject that which
is claiming conformance.
11.3 Checkpoint 11.3. Provide a conformance disclaimer. [Priority 3]
Conformance requirements: the specification MUST provide a conformance
disclaimer.
Conforms.
This is clearly fulfilled in Section 3.5.
11.4 Checkpoint 11.4. Impose no restrictions about who can make a claim or
where claims can be published. [Priority 1]
Conformance requirements: the specification MUST NOT include any
restriction about who can make a claim nor where claims can be published.
Conforms.
No restrictions are present.
Guideline 12. Publish an Implementation Conformance Statement proforma
12.1 Checkpoint 12.1. Provide an Implementation Conformance Statement
proforma.
[Priority 3]
Conformance requirements: the specification MUST either:
· provide an Implementation Conformance Statement.
· explain why an ICS is not needed
Conforms. The list of checkpoints is the ICS. This is explicitly stated
in Section 3.4
12.2 Checkpoint 12.2. Require the ICS be completed as part of the conformance
claim. [Priority 3]
Conformance requirements: the specification MUST include the ICS as part of
its conformance claim requirements. This checkpoint is not applicable, if
an ICS is not applicable to the specification.
Conforms. This requirement is also met in Section 3.4
Guideline 13. Clearly identify conformance requirements.
13.1 Checkpoint 13.1. Use conformance key words. [Priority 1]
Conformance requirements: the specification MUST use RFC 2119 keywords to
denote whether or not requirements are mandatory, optional, or suggested.
Conforms.
Done throughout by using MUST, SHOULD.
13.2 Checkpoint 13.2. Distinguish normative and informative text.
[Priority 1]
Conformance requirements: the specification MUST distinguish normative text
from informative text.
Conforms.
This is a big improvement from the last version I reviewed for conformance.
13.3 Checkpoint 13.3. Use consistent terminology. [Priority 1]
Conformance requirements: the specification MUST use identical wording to
express identical provisions, and analogous wording to express analogous
provisions .
Conforms, I guess.
However, it’s not clear to me exactly what this checkpoint is
requiring. What is meant, for instance, by “analogous wording to express
analogous provisions” and how can one test for this? This checkpoint is
pretty much unverifiable and untestable.
13.4 Checkpoint 13.4. Provide a fast way to find conformance information
[Priority 2]
Conformance requirements: the specification MUST provide at least one
navigation mechanism that allows the reader to locate by direct access, all
conformance-related information that is relevant to the specification. The
mechanism MUST minimally locate:
· the conformance section;
· unambiguous statements about those DoV that the
specification employs, from among the eight defined in this specification;
· requirements for conformance claims.
Conforms.
The Table of Contents satisfies this requirement.
Guideline 14. Provide test assertions.
Despite the claim that test assertions are present, I don’t believe that
the prioritized checkpoints fulfill the definition of test assertions. In
fact, the checkpoints are really the requirements, analogous to functional
requirements in recs.
14.1 Checkpoint 14.1. Provide test assertions [Priority 2]
Conformance requirements: the specification MUST provide a normative list
of test assertions stated in it.
Doesn’t Conform.
Although SpecGL specifically asserts that the test assertions are found in
the prioritized checkpoints, I don’t believe this meets the definition of
“test assertion”, as defined and explained in SpecGL. SpecGL defines test
assertion as “A test assertion is a statement of behavior, action or
condition that can be measured or tested. It is derived from the
specification's requirements and bridges the gap between the narrative of
the specification and the test cases. Each test assertion is an
independent, complete, testable statement for requirements in the
specification.”
The problem with using the checkpoints as fulfilling the test assertion
requirement are many. I’ll mention two: 1) Test assertions are supposed to
be “derived” from the spec’s requirements. The checkponts are the spec’s
requirements. 2) Some of these checkpoints can not be measured or tested
(see checkpoint 13.3 (analagous) as one example).
In addition, I do not believed that these checkpoints are specific enough
to be a “complete, testable statement”. See my previous write-up on Test
Assertions for guideline 3 at
http://lists.w3.org/Archives/Public/www-qa-wg/2002Dec/0104.html.
14.2 Checkpoint 14.2. Provide a mapping between the specification and the
test
assertions list [Priority 2]
Conformance requirements: the specification MUST provide a mechanism
linking each test assertion to the part of the specification it is stated.
N/A, since I don’t believe test assertions are provided.
No mapping or mechanism is provided.
Conclusion
This version is a big improvement over the previous version. However,
there are still some serious problems.
1. Guideline 8 is not met since SpecGL’s discretionary items (profiles,
modules, etc.) are not labeled as such and some do not include the required
rationale.
2. Guideline 9 (extensibility) has some problems. There is no uniform
mechanism to create extensions and SpecGL does not require that extensions
be published.
3. Guideline 11 is not met because specific wording of the conformance
claim has some missing elements.
4. Guideline 14 (test assertions) doesn’t conform
These are the areas that I said needed to be addressed during my last
review, with an annotation of whether or not it has now been addressed:
1 Rewrite of the narrative to eliminate ambiguity and delineate
requirements;
a. This has been done.
2. Categories and classes of products enumerated need to be updated to
include “specifications” (or “Framework Guideline”) as a category and a
product.
a. This has not been done.
3. An explicit statement saying “strict conformance in not required” needs
to be added;
a. No longer necessary due to re-write of SpecGL.
4. There should be a discussion of the circumstances (rationale for
inclusion) under when discretionary items (DOV) are allowed;
a. This has not been done.
5. SpecGL needs to discuss extensions and state that they are allowed and
then satisfy the other checkpoints in Guideline 9
a. For the most part this has been done. However, the checkpoints
addressing providing a uniform way to handle extensions and publishing them
not been met.
6. There needs to be a statement as to why an ICS is not needed or
(preferably) an ICS should be developed and included.
a. This has been done.
7. Some requirements need to be rewritten to be more specific and testable
(e.g., “analogous wording to express analogous provisions”)
a. This has not been done.
8. Test assertions need to be added (or SpecGL needs to say assertions are
not applicable to Guideline documents).
a. This has not been done.
****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970
Voice: 301-975-3262
Fax: 301-590-9174
Email: skall@nist.gov
****************************************************************
Received on Friday, 4 April 2003 09:45:14 UTC