When writing specifications it is critical to understand their primary purpose and scope. Clearly defined scope helps to keep the specification content focused and unambiguous.
The scope describes the areas covered by the specification, thereby indicating the intent or applicability of the specification. It does not specify requirements and is worded as a series of statements of fact.
The specification MUST define the subject matter of the specification, and SHOULD include a statement of objectives and/or goals. This information MUST appear at the beginning of the document.
The first section of the specification contains a clause entitled "scope" AND enumerates the subject matter of the specification.
The specification MUST provide examples or use cases illustrating what is in scope.
Test Assertion
The specification contains use cases to illustrate what is in scope OR the specification contains examples to illustrate what is in scope OR the specification contains use cases AND examples to illustrate what is in scope.
The specification MUST provide one or more examples of the functionality, concepts, and behaviors of the specification.
Test Assertion
For at least one functional requirement, concept or behaviour within the specification, the specification contains a corresponding specific example to clarify a complex concept, behavior or interaction.
Identifying what needs to conform to the specification is essential for the specification writers as well as for the implementors. This guideline relies on the concepts of classes of products and specification categories.
Classes of product is a dimension of variability (DoV).
A specification:
The specification contains a list of classes of products pertaining to this specification AND the specification defines conformance requirements for each product listed. If the list is NOT a proper subset of the classes defined in this QA Framework Guideline THEN the specification defines and describes the list of classes of products.
The conformance requirements indicate the conditions to be satisfied in order to claim conformance. It is likely that these conformance requirements will reference normative text within the specification or in other related specifications.
The specification MUST define the conformance requirements for each class of product identified in checkpoint 2.1.
Test Assertion
For each class of product identified in checkpoint 2.1, the specification contains a definition of conformance for each one.
The specification MUST identify in its introductory section the specification category or categories.
Test Assertion
In its introductory section, the specification contains a list of specification categories pertaining to this specification. If the list is NOT a proper subset of the categories defined in this QA Framework Guideline THEN the specification defines and describes the list of specification categories.
The specification MUST address and discuss the relations and interactions between classes of product and the other dimensions of variability.. This checkpoint is not applicable if there is only one class of product.
Test Assertion
If the specification contains AT LEAST two classes of products THEN the specification contains a section describing the relations and interactions between these classes of products and other dimensions of variability. If the specification contains ONLY ONE class of product THEN this checkpoint is not applicable.
To address various needs (re-usability, evolution, scaling, ...), technologies are often divided in different ways. These divisions have an important impact on the conformance to a specification. This guideline relies on the concepts of profiles, modules and levels that the QA WG has identified as being the three main division types in use.
Profiles, modules and levels are each a dimension of variability (DoV).
A specification
This checkpoint is not applicable if profiles are not used.
Test Assertion
If profiles are used by the specification THEN, for each identified class of product in the specification,he specification specifies EITHER that conformance is only defined within the context of prof the specification specifies EITHER that conformance is only defined within the context of profiles OR that conforming products may exist without respect to profiles AND describes any additional conditions associated with the profiles(s) for these class of products. If profiles are NOT used by this specification THEN this checkpoint is not applicable.
The specification MUST define for each profile the minimal required features/support for each identified class of product. This checkpoint is not applicable if profiles are not used.
Test Assertion
For each identified class of product the specification specifies the minimum required features supported by each profile for that class. If profiles are not used THEN this checkpoint is not applicable.
If the specification allows the creation of derived profiles, the specification MUST provide requirements for derived profiles and these requirements MUST be testable. This checkpoint is not applicable if profiles are not used.
Test Assertion
If the specification allows derived profiles THEN the specification contains testable requirements for these derived profiles. If the specification does NOT allow derived profiles THEN this checkpoint is not applicable.
The specification MUST document any identified mandatory conditions or constraints on the usage of modules. Such conditions include,
This checkpoint is not applicable if modules are not used.
Test Assertion
If the specification supports modules THEN the specification contains a section documenting all mandatory conditions or constraints associated with the use of modules.
The specification MUST address and document any identified relationships and interaction between profiles, modules, levels and with other dimensions of variability. This checkpoint is not applicable if none of the three DoV — profile, modules, and levels — are used.
Test Assertion
If the specification allow profiles OR the specification allow modules OR the specification allows levels THEN the specification documents all relationships and interactions among the profiles, modules and/or levels used with any other dimension of variability. If profiles are not used AND modules are not used AND levels are not used THEN the checkpoint is not applicable.
A deprecated feature is an existing feature that has been outdated by newer constructs or is no longer viable. Deprecated features should be avoided (e.g. for a format language, not be used by the authors and authoring tools) and may be removed in some future version, at which time the feature becomes obsolete.
After the initial publication of a specification, specification developers may consider the deprecation of a feature (e.g., function argument, element or attribute) defined in the specification. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecation of a feature may warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the effect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify different requirements for the support of deprecated features for each class of product. For example, authoring tools (producers) would not use the feature, but user agents (consumers) would continue to support it.
Deprecation is a dimension of variability (DoV).
The specification MUST document in a normative section each deprecated feature. This checkpoint is not applicable if there are no deprecated features.
Test Assertion
If deprecated features exist THEN the specification contains a normative section that documents each deprecated feature OR the specification contains a normative section containing a list of links to where the features appear in the document. If deprecated features are NOT used THEN this checkpoint is not applicable.
The specification MUST specify the degree of support required for each deprecated feature for each class of product and the conformance consequences of the deprecation. This checkpoint is not applicable if there are no deprecated features.
Test Assertion
If deprecated features exist THEN, for each class of product, the specification specifies the degree of support required for each deprecated feature AND specifies the conformance consequences of the deprecation.
The specification MUST indicate, for each deprecated feature, either (1) that it is independent of all other dimension of variability or (2) discuss the relationship between the feature and each of the other DoV.
Test Assertion
If deprecated features exist THEN, for each deprecated feature, the specification EITHER states that the deprecated feature is independent of all other dimensions of variability OR documents the relationship between the deprecated feature and each of the other DOV.
The specification MUST document each deprecated feature with a rationale for deprecation. This checkpoint is not applicable if there are no deprecated features.
Test Assertion
If deprecated features exist THEN, for each deprecated feature, the specification documents the feature AND includes a rationale for deprecation. If deprecated features do not exist THEN this checkpoint is not applicable,
The specification MUST provide an example for each deprecated feature showing how to avoid using it. This checkpoint is not applicable if there are no deprecated features.
Test Assertion
If deprecated features exist THEN, for each deprecated feature, the specification provides an example of how an alternative approach done in place of the deprecated feature will produce the same objective. If deprecated features do not exist THEN this checkpoint is not applicable.
The specification MUST document each obsolete feature. This checkpoint is not applicable if there are no obsolete features.
Test Assertion
If obsolete features exist THEN, for each obsolete feature, the specification provides documentation on the feature. If obsolete features do not exist THEN this checkpoint is not applicable.
Discretionary items are defined as deliberate and explicit grants of discretion by the specification, to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.
Discretionary items are often made available in specifications, to give implementers of the technology the opportunity to decide from alternatives when building applications and tools. Discretionary items may be considered necessary because of environmental conditions (e.g., hardware limitations or software configuration, or external systems), locality (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc.
Discretionary items come in three basic variants:
Discretionary items are a dimension of variability (DoV).
The specification MUST indicate the rationale for the discretionary items and MUST identify by labeling all discretionary items. This checkpoint is not applicable for specifications that do not have discretionary items.
Test Assertion
If discretionary items exist THEN the specification labels each discretionary item as such AND indicates the rationale for each discretionary item. If discretionary items do not exist THEN this checkpoint is not applicable.
The specification MUST describe any permitted variations or constraints for how the implementation dependent value or feature is realized by implementations.
If implementation dependent values or features exist THEN, for each implementation dependent value or feature, the specification describes any permitted variations or constraints for how the value or feature is realized by implementations.
The specification MUST indicate, for each identified discretionary item, whether zero, exactly one, or several of the choices/options are allowed to be implemented. If the allowable number is dependent on other dimensions of variability, the dependencies MUST be stated. This checkpoint is not applicable for specifications that do not use discretionary items.
Test Assertion
If discretionary choices exist THEN, for each discretionary choice, the specification indicates whether zero, one or several of the choices/options are allowed to be implemented. If the allowable number is dependent on other dimensions of variability, THEN the specification states the dependencies. If discretionary choices do not exist THEN this checkpoint does not apply.
The specification MUST document the identified policies for handling discretionary choices
Test Assertion
If discretionary choices exist THEN the specification identifies policies for handling them.
The specification MUST address and discuss the relations and interactions between deprecated features and all the other DoV.
Test Assertion
If discretionary items exist THEN the specification defines the relationship and interaction among discretionary items and all the other DOV.
An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Extensions can be implemented individually (e.g., one function at a time) or by defining new profiles, modules, or levels that incorporate additional functionality beyond what is defined in the specification.
Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Exercise caution in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whether extensions should be allowed.
Disallowing extensions for any part of the specification enables strict conformance: strict conformance is defined as conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented). Dimensions of variability (e.g., modules, profiles, levels) are not extensions if the specification defines them or allows them to be defined.
Extensions may be private (often vendor specific) or public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations.. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.
Specifications allow extensions for various reasons. Extensions allow implementers to include features that they consider to be required by their customers. Also, extensions often define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods.
Extensions are a dimension of variability (DoV).
The specification MUST state
and MUST document the rationale for allowing extensions by referencing use cases and/or project requirements.
Test Assertion
The specification states whether or not extensions are allowed. If extensions are allowed THEN the specification states the conditions under which these extensions are allowed and disallowed AND the scope of the extensions AND their effect on conformance claims AND any limitations or restrictions on the use of the extension AND documents the rationale for allowing extensions by referencing use cases and/or project requirements.
The specification MUST state that extensions cannot negate or change support for required functionality. This checkpoint is not applicable if extensions are not allowed.
Test Assertion
If extensions are allowed THEN the specification explicitly states that "extensions cannot negate or change support for required functionality". If extensions are not allowed THEN this checkpoint is not applicable.
The specification MUST provide a uniform way to define that extensibility is being invoked and MUST provide the syntax to be used to indicate the extension. This checkpoint is not applicable if extensions are not allowed.
Test Assertion
If extensions are allowed THEN the specification provides a uniform way to define that extensibility is being invoked and provides syntax to be used to indicate the extension. If extensions are not allowed THEN this checkpoint is not applicable,
The specification MUST require that the syntax and semantics of the extension be publicly documented. This checkpoint is not applicable if extensions are not allowed.
Test Assertion
If extensions are allowed THEN the specification explicitly requires that the syntax AND the semantics of the extension be publicly documented.
The specification MUST indicate via conformance requirements that implementations provide a mode under which they produce only conforming content. This checkpoint is not applicable if extensions are not allowed. This checkpoint is only applicable to specifications that identify producer of content as one of its classes of products.
Test Assertion
If extensions are allowed AND "producer of content" is one of the specification's classes of products THEN the conformance requirements within the specification contains a requirement that implementations provide a mode under which they produce only conforming content. ELSE the checkpoint is not applicable.
The specification MUST address and discuss the relations and interactions between extensions and all the other DoV.
Test Assertion
If extensions are allowed THEN the specification addresses and discusses the relations and interactions between extensions and all the other DoV.
The checkpoints of this guideline aim to ensure that normative content and conformance requirements are easily identifiable in specifications. Some of the desirable characteristics of conformance requirements are that they are mutually independent from other requirements, express a minimal requirement (i.e., are atomic), and are distinguishable and labeled (e.g., tagged). It is important that specifications provide unambiguous statements, that clearly define what is required in order to claim conformance and what is optional. It is important also that the conformance statements are easily identifiable as such, and easily locatable. Clarity is fostered by uniformity of structure and style, and consistency of terminology and phraseology.
There is a lot to be said for consistency and clarity within a document - it facilitates the understanding and comprehension of the document. Authors and editors of specifications should already be familiar with the Publication Rules (Member-only) [PUBRULES] and W3C Manual of Style [STYLE-MAN], which help to achieve clarity and uniformity. This guideline builds on those general document conventions, with a particular focus on the presentation and identification of conformance information.
The specification MUST identify conformance requirements:
Test Assertion
The specification identifies conformance requirements EITHER by using the RFC 2119 keywords to denote if the requirements are mandatory, recommended or optional OR (the specification defines an alternative way that conformance requirements are identified AND explicitly states why the RFC 2119 keywords were not used).
Normative content is the prescriptive part of the specification whereas informative content is for informational purposes and assists in the understanding or use of the specification.
The specification MUST distinguish normative text from informative content. .
Test Assertion
The specification clearly distinguishes between normative (prescriptive) content and informative (informational) content. The specification distinguishes between normative and informative examples, illustrations and use cases as well as text.
The specification MUST use identical wording to express identical provisions, and analogous wording to express analogous provisions.
Test Assertion
If two or more provisions in the specification are identical THEN the specification uses identical wording to express them AND if two or more provisions in the specification are analogous THEN the specification uses analogous wording to express them.
The specification MUST provide at least one navigation mechanism that allows the reader to locate by direct access, all conformance-related information that is relevant to the specification. The mechanism MUST minimally locate:
Test Assertion
The specification contains one or more navigation mechanisms that allows the reader to locate by direct access, all conformance-related information that is relevant to the specification. The mechanism specified locates the conformance section AND unambiguous statements about those DoV that the specification employs, from among the seven defined in this specification AND requirements for conformance claims.
A specification depends on another specification when it relies on or requires functionality (or behavior) from the other specification in order to work (function) correctly. This other specification provides a necessary condition or functionality.
The specification MUST have normative references to any identified specification it depends on and MUST describe the relationship between the specifications and any identified conformance implications
Test Assertion
If the specification depends on other specifications THEN the specification contains normative references to those other specifications AND describes the relationship between the specifications and any identified conformance implications.
A look at various W3C Technical Reports shows that the term "conformance" is often qualified, resulting in more than one type of conformance. It is important to convey an understanding of what is meant by conformance and how it applies to each class of product as well as each dimension of variability (e.g., modules) if applicable. For example, if the specification defines behavior for more than one class of product, there may be a separate conformance policy for each class. Similarly, if the specification defines modules, there may be a different conformance policy for each module.
In particular, a reader intending to implement a product in one of the product classes addressed by the specification should know what the WG wants for interoperability among products in the class. The developer should understand what forms, if any, of "product differentiation" are allowed among conformant products. The specification may need to explain how the rules apply and provide examples.
A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications.
The specification MUST include a normative section enumerating the minimal requirements that apply across conforming products of a class.
Test Assertion
The specification contains a normative section enumerating the minimal requirements that apply across conforming products of a class. ELSE there are no requirements for minimum functionality.
Any conformance concepts used in the specification MUST be defined, either by reference or by including the definition in the text.
Test Assertion
The specification contains a definition of new conformance concepts. ELSE the specification does not use any conformance concepts not defined in SpecGL.
A specification MUST justify each Dimension of Variability the specification uses.
Test Assertion
For each DOV the specification uses, the specification justifies the reason for its usage. ELSE no DOVs are used by the specification.
The specification MUST document its conformance policy in a dedicated section of the document.
Test Assertion
The specification contains a conformance section AND the conformance section documents the specification's conformance policy.
The specification MUST identify and define all the conformance designations used.
Test Assertion
If a single, monolithic (strict) conformance definition is not used THEN the specification identifies and defines all conformance designations (e.g., degrees, levels or categories of conformance).
A specification may differentiate conformance claims by designating different degrees or types of conformance in order to apply and group requirements according to modules, profiles, levels, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. The WG includes in the specification the way they want people to claim their conformance.
An Implementation Conformance Statement (ICS) provides a mechanism whereby a supplier of an implementation of the specification provides information about the implementation in a standardized manner. It is used to indicate which capabilities and options have been implemented, as well as the limitations of the implementation. An ICS usually takes the form of a questionnaire where product implementors report on the conformance of their product to the named specification.
An ICS is useful in disclosing optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements.
The basic and detailed information that an ICS provides can also be used to assess and deduce the interoperability potential of two or more products.
The specification MUST provide specific wording of the claim and the specific wording MUST at least include the specification name, its version, the conformance level satisfied and information about the subject that which is claiming conformance and the date of the claim.
Test Assertion
The specification contains specific wording relating to an implementation's claim of conformance. The specification includes, in this wording for the claim, the specification name AND the version, the conformance level satisfied AND information about the subject that which is claiming conformance AND the date of the claim.
The specification MUST provide a conformance disclaimer.
Test Assertion
The specification contains a conformance disclaimer that sends the message that a claim of conformanmance is no guarantee that the claimant is 100% conforming with the specification.
The specification MUST NOT include any restriction about who can make a claim nor where claims can be published
Test Assertion
The specification DOES NOT contain any wording that restricts who can make a claim AND the specification DOES NOT contain any wording about where claims can be published.
The specification MUST either:
Test Assertion
The specification contains an Implementation Conformance Statement OR the specification contains a discussion of why the ICS is not needed.
The specification MUST include the ICS as part of its conformance claim requirements. This checkpoint is not applicable, if an ICS is not applicable to the specification.
Test Assertion
If the specification contains an ICS THEN the ICS is referenced within the specification's conformance claim wording.
A test assertion is a statement of behavior, action, or condition that can be measured or tested. It is derived from the specification's requirements and provides a normative foundation from which test cases can be built.. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behavior. It is recommended that test assertions be available by the time a specification enters Proposed Recommendation.
The specification MUST provide a normative list of test assertions ..
Test Assertion
The specification contains a normative list of test assertions OR the specification references a separate document containing the test assertions. The test assertions cover all the requirements in the specification.
The specification MUST provide a mechanism linking each test assertion to the part of the specification it is stated.
Test Assertion
For each test assertion, the specification maps that assertion to the specific requirement(s) in the specification tested by the assertion.