Copyright ©2002 W3C ® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
This document defines a set of common guidelines for building conformance test materials for W3C specifications. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Materials Guidelines.
Note. This is a preliminary document plan - content only.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the first public Working Draft, and incorporates the discussions at the first QA face-to-face, plus several subsequent QA Working Group [QAWG] teleconferences, and supersedes all previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.
This part of the Framework document family will eventually have an accompanying "Testing Examples and Techniques" (in progress). Some of the lengthier examples and "how to" text of this current guidelines document version will be moved to the "Examples and Techniques" document.
Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.
1. Introduction
1.1 Navigating through this document.
1.2 Priorities
1.3 Terminology
2. Guidelines
G 1. Analyze the specification(s).
G 2. Define areas for testing
G 3. Choose the testing methodology
G 4. Provide the test automation and framework
G 5. Provide the results reporting framework
G 6. Organize tests development
G 7. Conduct testing
3. Relationship between QA and other WGs
4. Conformance
4.1 Conformance definition
4.2 Conformance disclaimer
5. Acknowledgments
6. References
7. Change History
The Guidelines in the document are organized chronologically. The document starts with the guidelines that apply at the formation of a Working Group (e.g., charter considerations) and continues leading the reader through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials. This document is applicable to all Working Groups, including those that are being rechartered or already exist. Working Groups may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work.
This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:
The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:
Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.
High quality and timely production of test materials are the key requirements to produce a high quality interoperable standard. Therefore each checkpoint has a priority level assigned by the QA Working Group based on the checkpoint's impact on the quality and timing of the test materials produced by a Working Group.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].
Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].
Guideline 1. Analyze the specification(s).
As with any product testing, the first step should be to analyze the subject. The better initial analysis can be done, the easier will be to design the test suite.
Checkpoint 1.1. Create a list of all the specifications used or referenced. [Priority 2]
[dd] I'd actually propose that this be priority 1, as interdependencies between specifications are vital to test, especially for API specifications and base specifications, such as XML, which is referenced by many others.
Most if not all of the specifications use notions and behaviors defined in other technical documents. Building the tree of the specifications being used helps to understand relationships between them, asses the testing work already done for the referenced specifications.
Checkpoint 1.2. Extract testable assertions from the target set of specifications. [Priority 1]
The QA Specifications Guidelines recommend to produce a set of testable assertions. Testable assertions serve as elementary formal statements
Checkpoint 1.3. Define those testable assertions that are part of conformance criteria [Priority 1]
Checkpoint 1.4. Extract all the discretionary behaviors defined in the specification [Priority 1]
Checkpoint 1.5. Define vague areas in the specification [Priority 2]
[dd] Purely editorial: I think that find vague areas in the specification is a better wording than define vague areas. Also, finding vague areas opens up the possibility to extract vague areas using a stylesheet, for example.
Checkpoint 1.6. List user scenarios for the specification [Priority 2]
[dd] Again, I'd go for moving this up to priority one. It is especially important for non-technical oriented specifications, such as WAI.
Guideline 2. Define areas for testing
1Checkpoint 2.1. Define target areas for testing [Priority 1]
Checkpoint 2.2. Prioritize testing areas [Priority 2]
Checkpoint 2.3. For each testing area, produce a set of sample testing scenarios [Priority 3]
Before creating test cases for certain area of the specification, it may be useful to design a set of sample testing scenarios, based on the user scenarios. Those are not actual tests, but rather examples. This helps to properly select testing framework, create templates for test cases, define future sub areas. Also, creating these test cases helps in mapping what areas of the specification will be tested given user-driven criteria.
Checkpoint 2.4. Map sample testing scenarios to testable assertions, discretionary behaviors and vague areas [Priority 3]
Guideline 3. Choose the testing methodology
[dd] This guideline covers the area of test framework, something I anticipate will be covered elsewhere. Here, only substantial issues relevant to incorporating existing frameworks, and not altering them, should be raised.
[dd] Available and applicable methodologies need to be given here.
Checkpoint 3.1. For each test area choose applicable testing approach [Priority 1]
[dd] As above, list available ones.
Checkpoint 3.2. Reuse publicly available testing techniques if applicable [Priority 1]
Guideline 4. Provide the test automation and framework
Checkpoint 4.1. Review available test frameworks, automation and adopt existing if applicable [Priority 1]
[dd] This checkpoint is partly incosisten with the immediately previous one; reordering them might help.
Checkpoint 4.2. Ensure the framework and automation are platform independent. [Priority 1]
Alternative is to provide an implementation of the framework for every platform
Checkpoint 4.3. Ensure the framework and automation are applicable to any product/content that implements the specification [Priority 2]
[dd] The two previous checkpoints could be given in one, stating that the testing framework chosen should, if possible, be platform independent. Also, we need to keep in mind that providing platform-specific test frameworks raises issues with adding work that needs to be done to the testing framework itself; if we were to provide platform-specific test framework, the QA WG or any party producing those would need to allocate time to produce and ascertain their quality.
Checkpoint 4.4. Ensure the framework makes it easy to add tests for any of the specification areas [Priority 2]
Checkpoint 4.5. Ensure the ease of use for the test automation [Priority 1]
Checkpoint 4.6. Ensure the framework allows for specification versioning and errata levels [Priority 2]
Checkpoint 4.7. Ensure the framework accounts choices allowed for discretionary behaviors in the specification [Priority 3]
Checkpoint 4.8. Ensure the framework allows for tests for optional behaviors defined in the specification [Priority 3]
[dd] Experience from the DOM TS shows that allowing for optional/multiple behaviours is a high priority on the wish list for the TS. Implementors want to be able to test particular behaviours as defined in the specification, especially as they may have chosen to support only parts of the specifications (eg. DOM builds on XML, which allows for entity expanding/entity preserving applications).
Checkpoint 4.9. Ensure the framework accommodates levels of conformance defined in the specification [Priority 1]
Checkpoint 4.10. Ensure the results verification is product independent [Priority 1]
Checkpoint 4.11. Ensure the framework allows to document the tests [Priority 2]
Checkpoint 4.12. Ensure the framework has proper test case management [Priority 3]
Checkpoint 4.13. Ensure the framework allows to measure specification coverage [Priority 2]
One effective way to measure the specification coverage is to map list of tests to the list of testable assertions grouped by areas.
[dd] Absolutely; this way of grouping works fine with the way we have discussed modules of specifications.
Guideline 5. Provide the results reporting framework
Checkpoint 5.1. Review available results reporting frameworks and adopt existing if applicable. [Priority 1]
Checkpoint 5.2. Ensure the results reporting is platform independent [Priority 1]
[dd] As above, given that the testing framework is uniform in functionality and independence, this will have been dealt with elsewhere.
Checkpoint 5.3. Ensure the results reporting is compatible with the test framework [Priority 1]
Checkpoint 5.4. Ensure the ease of use for results reporting [Priority 1]
Ensure the results reporting has sorting and filtering capabilities, etc.
Checkpoint 5.5. Ensure the results reporting allows for specification versioning and errata levels [Priority 2]
Checkpoint 5.6. Ensure the results reporting allows to export results in self-contained format suitable to publish on the web [Priority 2]
Checkpoint 5.7. Ensure the results reporting provides details on failures sufficient to investigate [Priority 3]
Checkpoint 5.8. Ensure the results reporting allows for history/storing analysis comments [Priority 3]
Guideline 6. Organize tests development
Checkpoint 6.1. Start with the test suite prototype and publish it. [Priority 2]
Checkpoint 6.2. Start with atomic tests first, according to priorities defined in Ck2.5 [Priority 1]
Checkpoint 6.3. Conduct regular public reviews of the test suite as specification and test suite development continues [Priority 2]
[dd] Ideally yes, but this will not necessarily the case if the TS will be produced within the WG
Checkpoint 6.4. Conduct regular specification coverage analysis. [Priority 2]
Guideline 7. Conduct testing
Checkpoint 7.1. A Working Group must publicly encourage conformance testing among vendors. [Priority 1]
[dd] And other interested parties.
A common practice is to support public discussion group dedicated to the test-suite, organize f2f meetings for vendors.
Checkpoint 7.2. Vendors to publish test results for their products. [Priority 3]
[dd] It may be that the W3C can have a special space where information pertaining to test results can be given, if not expliitly, then using links to those pages where the information can be found (in order not to have to provide disclaimers).
This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.
This section defines three levels of conformance to this specification:
A Working Group conforms to the "QA Framework: Operational Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.
To make an assertion about conformance to this document, specify:
Example:
"This QA processes and operations of this Working Group conform to W3C's 'QA Framework: Operational Guidelines', available at http://www.w3.org/TR/2002/qaframe-ops/, Level AA."
The checkpoints of this specification present verifiable conformance requirements about the operational aspects of Working Group quality processes. As with any verifiable test requirements, users should be aware that:
The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:
First draft outline