W3C home > Mailing lists > Public > www-qa-wg@w3.org > April 2003

Assessment of SpecGL's Conformance to SpecGL

From: Mark Skall <mark.skall@nist.gov>
Date: Fri, 04 Apr 2003 09:44:57 -0500
Message-Id: <5.1.0.14.2.20030404092618.01e5ba40@mailserver.nist.gov>
To: www-qa-wg@w3.org
To all,

Following is my assessment of SpecGL's present degree of conformance to 
itself.  At the end of the assessment I've included conclusions and an 
update of what has been fixed since my last assessment.  In summary, 
SpecGL's conformance is decidedly better than when I last reviewed it, but 
there are still some key areas where SpecGL does not conform.

Mark

Assessment of Conformance of SpecGL to SpecGL


QA Framework: Specification Guidelines W3C Working Draft 10 February 2003


Guidelines and Checkpoints

Guideline 1. Define Scope.
1.1     Checkpoint 1.1. Include the scope of the specification. [Priority 1]

Conformance requirements: the specification MUST define the subject matter 
of the specification, and SHOULD include a statement of objectives and/or 
goals. This information MUST appear at the beginning of the document

Conforms.

This is a big improvement over the last time I assessed conformance.  The 
wording is no longer vague.  One suggestion  Use the terms “normative” and 
“informative” (or “non-normative”) when describing the checkpoints, ExTech 
and the appendices.  This makes it clearer what is and isn’t required.


1.2     Checkpoint 1.2. Illustrate what is in scope. [Priority 2]

Conformance requirements: the specification MUST include examples or use 
cases illustrating what is in or out of scope.

Conforms.

ExTech provides examples and use cases.
1.3     Checkpoint 1.3. Provide Usage Scenarios. [Priority 3]

Conformance requirements: the specification MUST provide Usage Scenarios

Conforms.

ExTech provides examples and use cases.

Question  What is the difference between Checkpoint 1. 2 and 1.3 (i.e., 
what’s the difference between use cases and usage scenarios?)  Do we need 
both checkpoints?
1.4     Checkpoint 1.4. Provide examples. [Priority 2]

Conformance requirements: the specification MUST provide one or more 
examples of the functionality, concepts, and behaviors of the specification.

Conforms.

ExTech and narrative descriptions provide this functionality.
Guideline 2. Identify what needs to conform and how.
2.1     Checkpoint 2.1. Identify all classes of product. [Priority 1]

Conformance requirements: a specification:
               MUST list the classes of product it addresses
                MUST use the above enumerated classes names when they 
match those of the specification
                MUST define and describe identified product classes that 
does not match the enumerated set


Conforms.

Again, there is much improvement over the last time this requirement was 
reviewed.  The scope is much more precise and has a section addressing 
classes of products. “Technical Reports”, the class of product for SpecGL 
should probably be added to the enumerated list of classes of products 
described later in Guideline 2.

However, I’m not too sure about adherence to the second and third “MUST” 
above (especially the third).  Although the specific categories don’t apply 
to categorizing SpecGL, I believe, according to our requirements, there 
should be some reference to these categories in Section 1.2 and an 
explanation of SpecGL’s classes of products vis a vis the enumerated list 
(i.e, explicitly state that this is a new class of product). (Again, this 
could also be achieved by adding “Technical Reports” to the enumerated list 
in Guideline 2.)

2.2  Checkpoint 2.2. For each class of product, define the conformance
requirements. [Priority 1]

Conformance requirements: the specification MUST define the conformance 
requirements for each identified class of product.

Conforms.

The discussion on degrees of conformance (A, AA, AAA) in the conformance 
section clearly fulfill this checkpoint.

2.3  Checkpoint 2.3. Identify which of the categories of object are 
specified in
the document as a whole. [Priority 3]

Conformance requirements: the specification MUST identify in the 
Introductory section which of the categories of object are specified in the 
document as a whole.

Does not conform (but can be made to conform easily).

“Categories of objects” is still a poorly defined term.  They should be 
called “category of specifications”.  Although SpecGL does not fall within 
any of the categories enumerated, I believe this requirement says that 
SpecGL should explicitly state what “category” it is (Framework Guideline?).

2.4 Checkpoint 2.4. If there are several classes of products, define their 
relationships and interaction with other dimensions of variability. [Priority2]

Conformance requirements: the specification MUST address and discuss the 
relations and interactions between classes of product and the other 
dimensions of variability. It is not applicable if there is only one class 
of product.

N/A  Only one class of product.

Guideline 3. Specify conformance policy.


3.1  Checkpoint 3.1. Specify any universal requirements for minimum
functionality. [Priority 2]

Conformance requirements: the specification MUST include a normative 
section enumerating the minimal requirements that apply across all 
identified products of a class.

Conforms.

One can deduce from SpecGL’s conformance section that conforming to all 
priority 1 checkpoints are the minimal requirements.  However, it would be 
clearer if SpecGL said exactly that.
3.2     Checkpoint 3.2. If special conformance terms are used, include a 
definition in the specification. [Priority1]

Conformance requirements: the specification MUST be defined, either by 
reference or by including the definition in the text.

Conforms.  Terms like DOV and ICS are defined.
3.3     Justify any usage of a dimension of variability [Priority 1]

Conformance requirements: a specification MUST justify each Dimension of 
Variability the specification uses.

Conforms.  Extensibility is the only DOV used.  They are justified by the 
rationale in Section 3.2.

One can argue that A, AA, AAA are levels, but I don’t think they fit the 
intent of what we meant be DOV levels.

Guideline 4. Address the use of profiles to divide the technology.

Since SpecGL does not use profiles, this guideline does not apply.

4.1  Checkpoint 4.1. Indicate whether or not the use of profiles is mandatory
for conformance. [Priority 1]

N/A.


4.2  Checkpoint 4.2. If profiles are chosen, define any minimal requirements.
[Priority 2]

N/A.

4.3  Checkpoint 4.3. If profiles are chosen, define their relationships and
interaction with other dimensions of variability. [Priority 2]

N/A.

4.4  Checkpoint 4.4. If profiles are chosen, address rules for profiles.
[Priority 2]

N/A.


Guideline 5. Address the use of modules to divide the technology.

Since SpecGL does not use modules, this guideline does not apply.

5.1  Checkpoint 5.1. If modules are chosen, indicate any mandatory 
conditions or
constraints on their usage. [Priority 1]

N/A.

5.2  Checkpoint 5.2. If modules are chosen, define their relationships and
interaction with other dimensions of variability. [Priority 2]

N/A.


Guideline 6. Address the use of functional levels to divide the technology.

Since SpecGL does not use functional levels, this guideline does not apply.

6.1  Checkpoint 6.1. If levels are used, define their relationships and
interaction with other dimensions of variability. [Priority 2]

N/A.

Again, if we consider the degrees of conformance to be levels, SpecGL would 
conform since our section on Extensibility discusses the relationship to 
degrees of conformance A, AA, and AAA.

Guideline 7. Identify the relation between deprecated features and conformance.

Since SpecGL does not use deprecated features, this guideline does not apply.

7.1  Checkpoint 7.1. Identify each deprecated feature. [Priority 1]

N/A.

7.2  Checkpoint 7.2. For each class of product, specify the degree of support
required for each deprecated feature and the conformance consequences of the
deprecation. [Priority 1]

N/A.

7.3  Checkpoint 7.3. Include an explanation for the deprecation. [Priority 3]

N/A.

7.4  Checkpoint 7.4. Include examples to illustrate how to avoid using
deprecated features. [Priority 3]

N/A.

Guideline 8. Define discretionary items.

In many sections, especially here, it’s difficult to map rec type terms, 
like discretionary items, to SpecGL.  However, it seems like many of the 
DOV are discretionary items.  If so, SpecGL does not conform to these 
requirements.

8.1  Checkpoint 8.1. State the circumstances for when discretionary items are
allowed [Priority 1]

Conformance requirements: the specification MUST indicate the rationale for 
the discretionary items and MUST identify by labeling all discretionary 
items. It is not applicable for specifications that do not have 
discretionary items.


Doesn’t Conform.

Profiles, modules, levels, etc. are optional features and, thus, 
discretionary items. Although some rationale for inclusion can be inferred 
for profiles and levels, there is no rationale for the inclusion of 
modules.  Also, these are not labeled, in SpecGL, as discretionary items, 
as they are required to be.

8.2  Checkpoint 8.2. For implementation dependencies, address the allowable
differences between implementations [Priority 1]

N/A.

8.3  Checkpoint 8.3. Indicate any constraints on the number of choices/options
that can be implemented [Priority 2]

N/A.

8.4  Checkpoint 8.4. Promote consistent handling of discretionary choices.
[Priority 2]

N/A.
8.5     Checkpoint 8.5  If discretionary items are used, define their 
relationships and interactions with other dimensions of variability.

Conformance requirements: the specification MUST address and discuss the 
relations and interactions between profiles and all the other DoV.

N/A, although the checkpoint doesn’t match the conformance requirements.

The checkpoint asks to define relationships among DOVs if discretionary 
items are used (which they are in SpecGL).  However, the conformance 
requirements only address the relationships between profiles and DOV.

Guideline 9. Allow extensions or NOT

9.1     Checkpoint 9.1. Indicate if the specification is extensible. 
[Priority 1]

Conformance requirements: the specification MUST state the conditions under 
which extensions are allowed and disallowed.


Conforms.

Section 3.2 does an excellent job of explaining the extensibility of SpecGL 
and the specific requirements.

9.2     Checkpoint 9.2. If extensions are allowed, define their scope and 
constraints. [Priority 1]

Conformance requirements: the specification MUST state
               the conditions under which extensions are allowed,
                the scope of the extensions,
                their effect on conformance claims,
                any limitations or restrictions on the use of the extension
and MUST document the rationale for allowing extensions by referencing use 
cases and/or project requirements. It is not applicable if the 
specification does not allow extensions.

Conforms.

Again, Section 3.2 covers these items.  However, it’s not clear what is 
meant by the must requirement for “scope of the extensions” and “any 
limitations or restrictions on the use of the extension”.  I don’t think 
SpecGL addresses these requirements; however, it can be inferred that the 
“scope” is unlimited and there are no “limitations or restrictions”.


9.3  Checkpoint 9.3. Prevent extensions from contradicting the specification.
[Priority 1]

Conformance requirements: the specification MUST state that extensions can 
not negate or change support for required functionality. It is not 
applicable if extensions are not allowed.


Conforms.

Section 3.2 contains an explicit statement addressing this, as required.

9.4  Checkpoint 9.4. Define a uniform mechanism to create an extension. 
[Priority 3]

Conformance requirements: the specification MUST provide a uniform way to 
define that extensibility is being invoked and MUST provide the syntax to 
be used to indicate the extension. It is not applicable if extensions are 
not allowed.


Doesn’t conform.

SpecGL does not define a standard way to define extensions.

9.5     Checkpoint 9.5. Require that extensions be published. [Priority 3]

Conformance requirements: the specification MUST require that the syntax 
and semantics of the extension be publicly documented. It is not applicable 
if extensions are not allowed.

Doesn’t conform.

SpecGL does not require extensions to it be published.

9.6  Checkpoint 9.6. Require that implementations provide interoperable 
alternatives to extensions. [Priority 3]

Conformance requirements: the specification MUST indicate via conformance 
requirements that implementations provide a mode under which they produce 
only conforming content. This checkpoint is not applicable if extensions 
are not allowed.


Doesn’t conform.
SpecGL does not provide a mode to produce only conforming content.

9.7  Checkpoint 9.7  If extensions are allowed, define their relationships 
and interaction with other dimensions of variability.

Conformance requirements: the specification MUST address and discuss the 
relations and interactions between extensions and all the other DoV.

N/A since there are no other DOVs besides extensions.

Guideline 10. Provide a conformance clause.

10.1    Checkpoint 10.1. Include a conformance section. [Priority 1]

Conformance requirements: the specification MUST document its conformance 
policy and specific conformance requirements in a dedicated section of the 
document.

Conforms.

Section 3 (Conformance) is a very comprehensive conformance section.


10.2  Checkpoint 10.2. Make normative reference to specifications on which the
current specification depends [Priority 2]

Conformance requirements: the specification MUST have normative references 
to any identified specification it depends on and MUST describe the 
relationship between the specifications and any identified conformance 
implications


Conforms?

SpecGL makes a normative reference to RFC2119, upon which it depends for 
key word definitions and references RFC2119 in the conformance 
section.  Also, the references to ExTech have been rewritten to explicitly 
state that ExTech is informative, so that is no longer an issue.  However, 
Section 1.4 is a little fuzzy.  This section says that the QA framework 
documents are interrelated (thus they depend on each other?).  This Section 
should explictly state that there are no conformance inmplications with 
respect to the other documents.

Guideline 11. Specify how to make conformance claims.

Again, with minor changes, SpecGL can conform to this Guideline.

11.1  Checkpoint 11.1. Identify and define all conformance designations.
[Priority 1]

Conformance requirements: the specification MUST identify and define all 
the conformance designations used.


Conforms.

This is clearly done in Section 3.4 with the A, AA, AAA degrees of conformance.
11.2    Checkpoint 11.2. Provide specific wording of the claim. [Priority 3]

Conformance requirements: the specification MUST provide specific wording 
of the claim and the specific wording MUST at least include the 
specification name, its version, the conformance level satisfied and 
information about the subject that which is claiming conformance and the 
date of the claim.

Doesn’t conform.
SpecGL attempts to meet this requirement in Section 3.4 with the wording 
“To make an assertion about conformance to this document, specify: . . ”
SpecGL is missing the version and information about the subject that which 
is claiming conformance.
11.3    Checkpoint 11.3. Provide a conformance disclaimer. [Priority 3]

Conformance requirements: the specification MUST provide a conformance 
disclaimer.


Conforms.

This is clearly fulfilled in Section 3.5.

11.4  Checkpoint 11.4. Impose no restrictions about who can make a claim or
where claims can be published. [Priority 1]

Conformance requirements: the specification MUST NOT include any 
restriction about who can make a claim nor where claims can be published.


Conforms.

No restrictions are present.

Guideline 12. Publish an Implementation Conformance Statement proforma

12.1  Checkpoint 12.1. Provide an Implementation Conformance Statement 
proforma.
[Priority 3]

Conformance requirements: the specification MUST either:
               provide an Implementation Conformance Statement.
                explain why an ICS is not needed
Conforms.  The list of checkpoints is the ICS.  This is explicitly stated 
in Section 3.4



12.2  Checkpoint 12.2. Require the ICS be completed as part of the conformance
claim. [Priority 3]

Conformance requirements: the specification MUST include the ICS as part of 
its conformance claim requirements. This checkpoint is not applicable, if 
an ICS is not applicable to the specification.

Conforms.  This requirement is also met in Section 3.4


Guideline 13. Clearly identify conformance requirements.

13.1    Checkpoint 13.1. Use conformance key words. [Priority 1]

Conformance requirements: the specification MUST use RFC 2119 keywords to 
denote whether or not requirements are mandatory, optional, or suggested.

Conforms.

Done throughout by using MUST, SHOULD.
13.2    Checkpoint 13.2. Distinguish normative and informative text. 
[Priority 1]

Conformance requirements: the specification MUST distinguish normative text 
from informative text.

Conforms.

This is a big improvement from the last version I reviewed for conformance.
13.3    Checkpoint 13.3. Use consistent terminology. [Priority 1]

Conformance requirements: the specification MUST use identical wording to 
express identical provisions, and analogous wording to express analogous 
provisions .


Conforms, I guess.

However, it’s not clear to me exactly what this checkpoint is 
requiring.  What is meant, for instance, by “analogous wording to express 
analogous provisions” and how can one test for this? This checkpoint is 
pretty much unverifiable and untestable.

13.4  Checkpoint 13.4. Provide a fast way to find conformance information
[Priority 2]

Conformance requirements: the specification MUST provide at least one 
navigation mechanism that allows the reader to locate by direct access, all 
conformance-related information that is relevant to the specification. The 
mechanism MUST minimally locate:
               the conformance section;
                unambiguous statements about those DoV that the 
specification employs, from among the eight defined in this specification;
                requirements for conformance claims.


Conforms.

The Table of Contents satisfies this requirement.


Guideline 14. Provide test assertions.

Despite the claim that test assertions are present, I don’t believe that 
the prioritized checkpoints fulfill the definition of test assertions.  In 
fact, the checkpoints are really the requirements, analogous to functional 
requirements in recs.
14.1    Checkpoint 14.1. Provide test assertions [Priority 2]

Conformance requirements: the specification MUST provide a normative list 
of test assertions stated in it.


Doesn’t Conform.

Although SpecGL specifically asserts that the test assertions are found in 
the prioritized checkpoints, I don’t believe this meets the definition of 
“test assertion”, as defined and explained in SpecGL.  SpecGL defines test 
assertion as “A test assertion is a statement of behavior, action or 
condition that can be measured or tested. It is derived from the 
specification's requirements and bridges the gap between the narrative of 
the specification and the test cases. Each test assertion is an 
independent, complete, testable statement for requirements in the 
specification.”

The problem with using the checkpoints as fulfilling the test assertion 
requirement are many.  I’ll mention two: 1) Test assertions are supposed to 
be “derived” from the spec’s requirements.  The checkponts are the spec’s 
requirements. 2) Some of these checkpoints can not be measured or tested 
(see checkpoint 13.3 (analagous) as one example).

In addition, I do not believed that these checkpoints are specific enough 
to be a “complete, testable statement”.  See my previous write-up on Test 
Assertions for guideline 3 at 
http://lists.w3.org/Archives/Public/www-qa-wg/2002Dec/0104.html.

14.2  Checkpoint 14.2. Provide a mapping between the specification and the 
test
assertions list [Priority 2]

Conformance requirements: the specification MUST provide a mechanism 
linking each test assertion to the part of the specification it is stated.

N/A, since I don’t believe test assertions are provided.

No mapping or mechanism is provided.




Conclusion

This version is a big improvement over the previous version.  However, 
there are still some serious problems.
1. Guideline 8 is not met since SpecGL’s discretionary items (profiles, 
modules, etc.) are not labeled as such and some do not include the required 
rationale.

2. Guideline 9 (extensibility) has some problems.  There is no uniform 
mechanism to create extensions and SpecGL does not require that extensions 
be published.

3. Guideline 11 is not met because specific wording of the conformance 
claim has some missing elements.

4. Guideline 14 (test assertions) doesn’t conform


These are the areas that I said needed to be addressed during my last 
review, with an annotation of whether or not it has now been addressed:
1  Rewrite of the narrative to eliminate ambiguity and delineate 
requirements;
a.      This has been done.

2. Categories and classes of products enumerated need to be updated to 
include “specifications” (or “Framework Guideline”) as a category and a 
product.
a.      This has not been done.

3. An explicit statement saying “strict conformance in not required” needs 
to be added;
a.      No longer necessary due to re-write of SpecGL.

4. There should be a discussion of the circumstances (rationale for 
inclusion) under when discretionary items (DOV) are allowed;
a.      This has not been done.

5. SpecGL needs to discuss extensions and state that they are allowed and 
then satisfy the other checkpoints in Guideline 9
a. For the most part this has been done.  However, the checkpoints 
addressing providing a uniform way to handle extensions and publishing them 
not been met.

6. There needs to be a statement as to why an ICS is not needed or 
(preferably) an ICS should be developed and included.
a.      This has been done.

7. Some requirements need to be rewritten to be more specific and testable 
(e.g., “analogous wording to express analogous provisions”)
a.      This has not been done.

8. Test assertions need to be added (or SpecGL needs to say assertions are 
not applicable to Guideline documents).
a.      This has not been done.










****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
**************************************************************** 
Received on Friday, 4 April 2003 09:45:14 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:13 GMT