Version 0.3
November 21, 2001
Editors:
Lynne Rosenthal (lynne.rosenthal@nist.gov)
Contributors:
Mark Skall (mark.skall@nist.gov)
Lofton Henderson (lofton@rockynet.com)
David Martson (david_marston@lotus.com)
Abstract
Document identifying the conformance requirements that need to be included
or addressed in specifications. Target audience is primarily specification
developers, followed by conformance test suite developers and implementation
developers
Status of this Document
Pre-Note -- this document will be formatted as a W3C Note.
As such it will serve as a contribution towards the QA Specification Framework,
addressing some but not all of the topics of the latter. It is not intended
to be the Specification Framework document but to remain as a Note, providing
supplemental information that is focused conformance section and statements
in a specification.
Document Version History
21 Nov 2001
updated based on input from QAWG
Oct 2001
input from OASIS Conformance TC
Reference Documents
ISO Guide 2: Standardization and related activities -- General vocabulary.
ISO/IEC Directives Part 3: Rules for the structure and drafting of International
Standards
ebXML Technical Architecture
Specification, Conformance Clause
W3C WAI Guidelines
UNICODE
The objective of this document is to identify the conformance requirements that shall be included or addressed in specifications. Conformance requirements are the expression, in the form of a statement, which conveys the criteria to be fulfilled [ISO Guide 2]. The conformance requirements are stated in a conformance clause or statements within the specification. Note that additional conformance information and guidance may be contained in supplemental documentation. This document describes the purpose and scope of a conformance clause as well as the associated issues that a conformance clause should address. Where ever possible, sample text and examples will be given.
The information contained is produced as the result of extensive experience in the development and implementation of conformance clauses and test suites for consensus standards and specifications. It is based on the principles and requirements prescribed by international standards (e.g., ISO/IEC and IEEE) as well as extractions from ebXML, OASIS and W3C specifications.
This document specifies the general requirements and definitions concerning conformance and related issues. It is intended to contribute fundamentally towards mutual understanding amongst developers of specifications and conformance test suites and tools. It is also intended to provide a suitable source for teaching and for reference, briefly covering basic theoretical and practical principles of conformance.
This document will not define specific conformance requirements for any specific specification -- this is the responsibility of committees chartered to develop functional specifications.
This document is intended primarily for the developers of specifications to help enable them to develop conformance requirements within their specification and to create a testable, unambiguous specification. Secondary audiences include, but are not limited to: developers of conformance test suites, software implementers, international standards bodies, and other industry organizations.
In order to conform to this Conformance Requirements document, a specification shall contain a conformance clause. It is recommended that the conformance clause exist as a separate section within the specification, so that it is clearly identifiable. A specification that conforms to this document shall:
For each issue the specification shall be clear whether or not the issue is addressed and its disposition. For example, if a specification does not contain levels it should be clear to the reader that levels are not supported. One method to ensure this clarity is to explicitly state that levels are not supported.
[Editor note: we need to make sure that this conformance clause conforms to this document - add what is needed to address all issues in Section 8, etc.
The following normative documents contain provisions, which through reference in this text, constitute provisions of this document. At the time of publications, the editions indicated below were valid. All standards are subject to revision, and parties to agreements based on this document are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below.
ISO/IEC Guide 2: Standardization and related activities -- General vocabulary
ISO/IEC Directives Part 3: Rules for the structure and drafting of International Standards.
RFC 2119: Keywords for use in RFC's to Indicate Requirement Levels
World Wide Web Consortium Process Document
For the purposes of this document, the following relevant terms and definitions apply:
Certification -- the acknowledgement that a validation has been completed and the criteria established by the certifying organization has been met.
Conformance -- the fulfillment of a product, process, or service of specified requirements.
Conformance Testing -- a method of verifying implementations of a specification to determine whether or not deviations from the specification exist.
Implementation -- the realization of a specification -- it can be a software product, system, program, protocol, application, or document instance.
Validation -- the process of testing software for conformance to a specific specification.
The conformance clause is a section of a specification that defines the requirements, criteria, or conditions that must be satisfied to claim conformance. The conformance clause identifies what must conform and how conformance can be met. Typically it is a high-level description of what is required of implementers and applications. It may refer to other parts of the standard. It may specify sets of functions, which may take the form of profiles, levels, or other structures. Additionally, it may specify minimal requirements for certain functions and for implementation-dependent values.It may also specify the permissibility of extensions, options, and alternative approaches and how they are to be handled.
Every specification shall contain a conformance clause.
A conformance clause:
There are specific words that are used throughout the specification to denote whether or not requirements are mandatory, optional, or suggested.; Using these keywords help to identify the testable statements in a specification.; Although the keywords used within the ISO/IEC community differ from the keywords used within the IETF communities, they achieve the same results.; Use of these keywords should be consistent (i.e., use the ISO keywords or the IETF keywords, but do not use both).
ISO Keywords:
IETF Keywords ([RCF2119]):
Additional keywords include:
An objective of any conformance clause and its related conformance statements is to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional. To achieve this objective,
The conformance clause identifies the "class of products" (i.e., object of the claim) that will be developed, where "class of product" may be an implementation, application, service, and/or protocol (e.g., content, user agent, authoring tool). Additionally, the clause specifies the conditions that shall be met in order to claim conformance for that class of product (i.e., make a valid claim). It may also specify that which is not a requirement. There may be several classes of products that are identified, each with its own conformance statement or set of conformance criteria.
For example, the W3C DOM Recommendation defines a DOM implementation or host implementation (e.g., a browser) and a DOM application or client application (e.g., a script which runs in a browser).
A class of product may consist of several integrated components rather than a single piece of software (e.g., browser). Conformance may be defined in terms of the integrated software (system) and/or for each component. Any restrictions or constraints on the number or types of components that make up the "subject of a conformance claim" shall be specified.
For systems that are comprised of several components, it may be sufficient to state that conformance to the system is equivalent to conformance of all the required components considered individually, and the system satisfies at least the minimum conformance requirements for each of those components.
For example, the conformance clause in the ebXML Technical Architecture states, "ebXML conformance is defined as conformance to an ebXML system that is comprised of all the architectural components of the ebXML infrastructure and satisfies at least the minimum conformance requirements for each of the ebXML technical specifications."
A specification may apply and group requirements into various conformance levels. These levels may relate to functionality, impact and/or incremental degrees of implementation. The conformance clause shall identify and define all conformance levels.
For example: The W3C Web Accessibility Guidelines define three levels of conformance (Level A, Double-A and Triple A) based on the checkpoint priority levels satisfied. Conformance Level A: all Priority 1 checkpoints are satisfied; Conformance Level Double-A: all Priority 1 and 2 checkpoints are satisfied; and Conformance Level Triple-A: all Priority 1, 2, and 3 checkpoints are satisfied.
The specification may provide the specific wording of the claim (appendix A provides sample conformance claims). It may also require specific information be contained in the claim, such as name/date/version of the specification, test suite, and tested product.
The specification shall impose no restrictions about who can make a conformance claim (e.g., vendor, user, third party) or where the claims may be published. It may provide additional information regarding the responsibility of claimants.
Often implementations do not use all the features within a specification. In order to accommodate these implementations it may be desirable to divide a specification into sets of functions. Implementors would be allowed to implement one or more of these sets rather than the entire standard. These sets are commonly implemented as profiles or levels.
Profiles are used as a method for defining subsets of a specification by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users. Specifications that explicitly recognize profiles should provide rules for profile creation, maintenance, registration and applicability. Appendix A provides additional information on profiles.
Levels are used to indicate nested subsets of functionality, ranging from minimal or core requirements to full or complete functionality. Typically, level 1 is the minimal or core of the specification that must be implemented by all products. Level 2 includes all of level 1 and also additional functionality. This nesting continues until level n, which consists of the entire specification.
It is possible for a specification to have both profiles and levels. If profiles and/or levels are defined, the conformance clause specifies which (if any) of these profiles and/or levels is mandatory. Additionally, any conditions associated with a particular profile, level or combination of these needs to be specified.
Experience has shown that having too many profiles and/or levels can inhibit interoperability as well as add confusion to the marketplace.
If profiles and/or levels exist, the specification shall indicate the conditions for claiming conformance to a specific profile and/or level. In particular, consider whether or not a claim of conformance to a particular profile/level can include functionality or features of a higher profile/level. Typically, implementations that purport to conform to a specific level n of a specification may include functionality defined within one of the higher levels n+1. The specification requirements for extensions (Section 8.2.2) are applicable for profiles/levels and should be adhered to.
An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Allowing extensions affects how conformance is defined as well as what conformance claims could be made. Care should be exercised in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whether extensions should be allowed in conforming content or implementations. It may be better to stipulate that the presence of extensions is non-conforming, than to have content or implementations which cannot interoperate but which may be claimed as conforming to the specification. Appendix C provides additional information about extensions.
If a specification disallows extensions, then the conformance clause shall specify that extensions are not allowed and that implementations of the specification shall precisely implement the complete specification. This is called strict conformance. Strict conformance is often imposed on applications (i.e., content) of a specification. For example, a software program or XML document instance. Strict conformance may also be imposed on implementations (e.g., as in Ada). Note, that this prohibition of extensions could be applied to a specific profile or level rather than to the entire specification.
If specification allows extensions, then the conformance clause shall state the conditions under which extensions are allowed, the applicability of the extensions, their affect on conformance claims, and any limitations or restrictions on the use of the extension.
The conformance clause should require that each implementation shall fully support all required functionality of the specification exactly as specified and that the use of extensions shall not contradict nor cause the non-conformance of functionality defined in the specification. Additional requirements may include:
In some instances, it may not be possible to define the behavior or values of a function. Implementation dependent means that an implementation may determine the effect (rather than having the effect mandated by the specification). However, such effects shall be consistent within a single implementation (e.g., a browser's rendering of ??? shall be the same for every invocation regardless of the document instance).
Details in a specification may deliberately be omitted (i.e., not specified), so as to provide freedom to adapt implementations to different environments and different requirements. In general this is not a recommended practice. Caution should be exercised if details are omitted and used only in a limited number of instances.
Specifications shall indicate implementation dependencies and where applicable, address allowable differences between implementations including,
Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats, protocols, or codes). In such a case, the conformance clause shall specify under what conditions an implementation is considered to be conformant. Some possible ways to define conformance include mandating that an implementation shall:
Note: if the specification doesn't describe the different approaches, this becomes an implementation detail irrelevant to conformance.
Every specification shall identify, either by default or explicitly, a single natural language or a more formal specification language (e.g., IDL, UML) edition as the normative version.
Every specification shall specify whether it permits multiple or alternative natural languages, language bindings and/or character encodings. If it does, it shall specify the languages and encodings that must be supported by conforming implementations. Additionally, the error conditions and/or behavior to handle situations in which unsupported languages or encodings are encountered shall be defined.
When specifying characters, the Unicode Standard [ISO 10646] shall be used.
A specification may include an Implementation Conformance Statement (ICS) or questionnaire and require its completion as part of a conformance claim. An ICS is useful in clarifying and declaring optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against the relevant requirements and against those requirements only. The ICS is also helpful in describing the expected interoperability to be achieved with other implementations or applications of the specification.
If an ICS proforma is included as part of the specification, it shall be explicitly identified as either a normative or informative part of the specification.
Anything to add from the XSLT work?
A specification may include test assertions as part of the specification. A test assertion is a statement of behavior, action or condition that can be tested or measured. It is derived from the specification requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for a requirement in the specification. Each test assertion results in one or more test cases.
Including test assertions as part of the specification facilitates and promotes the development of conformance test suites and tools. Specific benefits include:
Including test assertions as part of the specification has been done is several IEEE and ISO standards including IEEE POSIX and ISO 10303 (STEP).
A specification may provide a test framework, methodology and/or procedures for testing to the specification. This type of information ensures consistency between testing programs and organizations, and provides confidence in those testing programs. If any of this information is provided, it shall be explicitly identified as either normative or informative guidelines.
The test method may describe the conformance testing approach – the use of methods involving rigorous proofs of correctness in which conformance can be conclusively and exhaustively demonstrated (e.g., the syntactic validators for HTML, CSS, WAI content) or the use of methods involving falsification testing.
The test methodology may describe the different types of conformance tests and tools that need to be developed, the type of test materials that need to accompany the tests, and the type of information that must be contained in a test report.
The procedures for testing may describe the organizational structure, activities and responsibilities for external organizations that establish and operate a testing service for the specification.
The procedures for testing may prescribe how testing is conducted, e.g., self-declaration or third party testing laboratories. It may also provide a step-by-step guide for using the tests or tools correctly so that the results may be repeatable and reproducible.
This type of information is provided as normative sections in several standards, e.g., ISO 10303 (STEP) and ISO 15046 (Geographic Information), and as part of several consortia specifications.
(Informative)
In general, a conformance claim should contain the name and version of the tested implementation, the specification, name and version of the test suite, date testing was completed, conformance level (or profile) satisfied, and the results of the testing. For example:
Name of Implementation and version has been tested for Level L conformance to Name of Specification using the Name of Test suite, ver X.X on YY-MM-DD and no nonconformities were found.This Name of Implementation (fully specified) has been tested for conformance to Name of Specification, in accordance with the XXX Validation Procedures using the Test Suite and testing environment listed below:-Name of Certificate Holder:-Implementation Identification:-Testing Environment (hardware/software):-Test Suite name and version-Level of Conformance:-Nonconformities:-Test Report: provide a URI
The Web Content Accessibility Guideline requires a claim to contain the title of the guidelines document, its URI, the conformance level satisfied, and the scope covered by the claim (e.g., page, site), for example:
This page conforms to W3C's "Web Content Accessibility Guidelines 1.0", available at http://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505, level Double-A
(Informative )
The following is extracted from ISO 8632 Computer Graphics Metafile Standard
A profile of a specification defines the options, elements, and parameters necessary to accomplish a particular function and maximize the probability of interchange between systems implementing the profile. Profiles are defined to meet the requirements of application constituencies who are asked to adhere to the same subset of the specification. A profile may be a subset of a single specification or may be part of the set of interrelated standards and profiles assembled for the purpose of accomplishing a larger functional purpose. A profile shall not specify any requirement that would contradict or cause non-conformance to its specification.
A profile may:
Profiles provide a means to:
An extension may be private (often vendor specific) or may be public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc are defined and published allowing anyone to implement the extended functionality.
[Ed NOTE – need to clean up this section]
One mechanism to allow extensions within a specification is to provide a standard way of defining the extension. Basically, this provides a "standard way of being non-standard". This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate action (i.e., to ignore or to implement). The nature of the extension may dictate the method for defining the extension. It may be possible to define a function that indicates external (from the specification) functionality. This external function may take the form of an escape or control character or be an identifier, which whenever invoked indicates an extension follows. Another method, especially when extending a list of numeric parameters is to use a scheme where positive values represent standardized values and negative values are reserved for private use.
For example, the ISO Computer Graphics Metafile (CGM) is the standard on which the W3C WebCGM Recommendation is based. It provides both a standard function (GDP element) for defining private graphics functionality, as well as the use of negative values to define private values.
Another mechanism that minimizes interoperability problems when extensions are allowed is to have a register for extensions. This document, distinct from the official specification, contains a list of recognized extensions to the standard. See section below.
Registration is a procedure that allows extensions to be acknowledged and made available to the public. The procedures provide for the extension to have some rigor and technical review. Typically, the committee developing the specification is responsible for processing the registration of an extension, thus ensuring adequate quality of a proposed extension and a technical description sufficient to be uniformly implementable. Often registered extensions may migrate into a later version of the specification