Abstract
Tests for software can be written in many different ways, varying in quality and efficiency. One typically needs to invest more time and effort the more complete the test suite needs to be. In addition, test suites are sometimes equal in difficulty with the software being tested. Aiming at helping produce conformant software is especially relevant for W3C WGs (as well as other bodies). The benefit of using a well thought out test framework translated to less effort, more time to concentrate on producing good software and reuse of existing frameworks. The goal of this document is to present those guidelines and principles in way that is easy to implement. The document is presented as a set of guidelines or principles, supplemented with good practices, examples, and techniques.
Status of This Document
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. A list of current
W3C publications and the latest revision of this technical report can be
found in the W3C technical
reports index at http://www.w3.org/TR/.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity
statement.
This is the first published version of this document after the decision to
completely redesign the QA Framework (QAF), resolved by the QA Working Group (QAWG) at its 2004 Technical Plenary
face-to-face. It is a lighter-weight, less authoritarian, more
user-friendly and useful version of the previously published Working Draft version of the Test Guidelines
This draft accurately reflects the intended overall structure and content
of the new, lightweight Test Guidelines. However it is
incomplete in some details (generally indicated by "@@" or "TBD"), especially in the examples -- lots more
examples are intended. Also, some sections more resemble outlines than
completed prose. Because this is a significant departure from the
last-published Test Guidelines, QAWG wants review and comment on the
overall direction as soon as possible. Incomplete details will be fleshed out
in the next publication.
This Test Guidelines document will be coordinated
with the other parts of the QA Framework, The QA Handbook [QA-HANDBOOK] and
QA Framework: Specification Guidelines. Synchronization at uniform level
of maturity will occur no later than Last Call. The first publications of the
various parts will occur somewhat more independently.
The QAWG expects this
specification to become a Recommendation.
Publication as a Working Draft does not imply endorsement by the W3C
Membership. This is a draft document and may be updated, replaced or
obsoleted by other documents at any time. It is inappropriate to cite this
document as other than work in progress.
This document has been produced under the
24 January 2002 CPP as amended by the
W3C Patent Policy
Transition Procedure. An individual who has actual knowledge of
a patent which the individual believes contains Essential Claim(s)
with respect to this specification should disclose the information
in accordance with section 6 of the W3C Patent Policy.
Patent disclosures relevant to this specification may be found on the
Working
Group's patent disclosure page.
Comments on this document may be emailed to www-qa@w3.org, the publicly archived list
of the QA Interest Group
(QAIG).
Commenters please note that your comments will be publicly
archived and available, do not send information that should not be
distributed, such as private data. If you are able to give your feedback by
12 June 2004, the QAWG can
consider it at its next face-to-face meeting. The comments will still be collected after the F2F in June 2004.
Table of contents
REBUILD
Introduction
This document defines principles & practices to support the creation of
useful and usable conformance test-suites. While much of its contents
are applicable to other forms of testing, the scope of this revision of the
guidelines is limited to conformance testing. In particular, it follows directions pointed out in other QAF documents @@ QAH and SpecGL. The document also mentions other types of testing that can be used to derive good ideas and practices; however the primary scope is that of conformance testing.
Scope and Goals
This document is a guide for W3C specification editors and test authors. It provides
guidelines for writing better tests and test frameworks for W3C specification. It aims at providing simple, intuitive and easy to follow principles worth implementing when designing and releasing a Test Suite (TS) aimed at conformance testing.
There are relations with other QA techniques, sich as writing implementable specifications aimed at conformance. These relations will be dealt with in the document.
The goal is to make test suites usable to implementors when assessing conformance to a specification. Furhtermore, the document aims at, in so far as is possible and valuable, streamline test production process to make it easier to implement for WGs.
Motivation
Given a well written specification, the next natural step is to write implementations that conform to it. The only wasy to do this successfully, is to test the implementation against the specification. Since there are reasons for providing a fairly uniform set of specification authoring principles, the QA WG believes the same holds for the test frameworks used to assess conformance.
It is in everyone's interest that test suites be of as high quiality as possible, and especially that their quality and efficiency be generally accepted, especially since this simplifies for implementers to write conformance software.
Given the requirements in the specification, a test author or test suite framework producer is interested in an easy way to produce tests, and an equally easy way to maintain the tests as well as gather feedback on them, for the end result to be as good as possible.
Why Test Guidelines?
@@Add wording
About this document
@@Add wording
Audience of this
document
@@Add wording
The primary audience of this document is builders of test materials and conformance test suite and tools producers, however, it
is applicable to a broader audience including:
- those who review test during their development
- specification authors and implementers.
Structure of this document
This document is organized in a series of principles, each of which contains a number of requirements that are further explained by examples and, where applicable, techniques or process issues. Techniques and process issues are fairly detailed, whereas principles are general in nature.
Does this division hold or should the SpecGL main topic - principle - best practice be used instead?
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY ", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they will be all uppercase. Occurrences of these words in lowercase comprise normal prose usage, with no normative implications.
This document can be used as a stand alone document or as part of a family
of QA Framework documents designed to help the WGs improve all aspects of
their quality practices. The QA Handbook [QA-HANDBOOK] is a non-normative handbook about the process and operational aspects of the quality practices of W3C's Working Groups (WG). @@Add reference to the Spec GL@@
Principles
Test suite scope
Both test builders and test suite producers, as well as implementers, need to understand the scope of the test suite. It is essential in order to know whether the test suite is applicable to them, the extent to which the test suite results are relibalbeand it finally serves as an indicator or where efforts need to be invested.
Requirement: Document the scope and goals of the test suite.
In order to have a clear picture of what is covered in the test suite, specify what specifications and parts thereof are covered. Also indicate what testing strategy was adopted when developing tests. Finally, mention the goals of the test suite (this document mainly deals with conformance testing).
Examples
Indicate whether, for example, tests have been written in parallel with the specification or if they have been written after speification completion (or some degree of maturity).
Requirement: Provide coverage information.
Test suite users need to know what is covered and what is not, especially if test suites are used to assess conformance. It must be possible to map test back the specification; if a test fails, the test suite user must understand what part of the implementation is at fault.
Should must here be RFC MUST?
Examples
Provide a coverage map, pointing from each test back to the part of the specification it was written for.
Good Practice: Assertion lists are an effective way of documenting tests and mapping them back to the specification.
Test execution results must be repeatable and reproducible.
If test results are not repeatable and reproducible they are not reliable, and they cannot be compared with other test suite execution results. [@@ include pointer to definition of repeatable and reproducible @@].
Requirement: Define the contents of the test suite.
The components of a particular version of a test suite must be unambigously identified. It is not sufficient to point users to a randomly updated web site that contains an amorphous collection of test materials. Test materials must be packaged together into a "test suite" and published with a version number, indicating any changes from previous versions. The test suite must, in addition, contain documentation that describes its contents and explains how to use it.
Potential components of a test suite:
- Tests (may be "object", or it may be "source" that needs to be processed to generate "object" (an analogy is between that of source and executable code, respectively).
- Test materials metadata [@@ needs to be defined] sufficient to allow [@@ define intended functionality].
- Test harness (optional)
- Supporting libraries and tools
- Documentation
- Conformance requirements
Requirement: Specify what tests are to be run.
It must be possible to determine what tests should be executed for a particualr implementation, allowing non-applicable tests to be filtered out. This is particularly important for specifications that define modules, or for implementations that explicitly do not support some part of a specification (it may help to think of validating and non-validating XML parsers).
Good Practice: defining metadata enables filtering.
Requirement: Document how to execute tests.
Test suite documentation must clearly explain how to execute the tests.
Good Practice: either provide a test harness and supporting tools, libraries and framework, or provide sufficient metadata and documentation to allow a test harness to be constructed.
Requirement: Tests should report their status in a meaningful and consistent manner.
Tests should report execution status (passed, fail, not run etc [@@ need to define these] unambigously and consistently.
Good Practice: If possible, tests should report what went wrong (what the expected results was and what actually happened), as an aid to debugginf the problem.
Requirement: Ensure repeatability and reproducibility.
Test suite execution must be the same for two users with identical setup (all other things equal).
Good Practice: Before release, conduct extensive test suite execution to make sure test suites and indeed repeatable and reproducible.
Test suites must evolve over time
Test suites must evolve over time, as problems are identified and fixes, as coverage is increased or to adress errata and revisions of applicable speficications. In short, test suites and frameworks must have versioning.
Requirement: plan for multiple releases.
Plan for multiple releases of the test suite. Ideally, a new version of the test suite should be released for each revision/errata of the specification. Version numbers and relevant documentation should be supplied. Users should understand which version of the test suite is appropriate for a particular implementation.
Good Practice: Plan for and implement a versioning system.
Requirement: Accept and respond to bug reports.
In order for a test suite to meet the goals, it is good to have tests reviewed by as many people and implementers as possible. For this to happen, plan to implement a bug reporting mechanism.
Good Practice: Implement a formal bug- or issue-tracking system to manage bug reports.
Users must be provided with a formal channel for reporting problems in the test materials (tests, test harnesses, documentation). Problems in the tests may reveal problems in the specification, especially interpretation issues. Problems can be adressed by:
- Excluding broken tests from the test suite
- Creating and distributing alternate tests
- Updating documentation, harness, framework etc
- Modifying the specification
Good Practice: Patching test suites is difficult to do, instead, consider re-releasing the the test suite under a new version. This way, confusion is minimized.
Requirement: Maintain test suite beyonf WG life
Implementations are written for specifications that do not longer hanve chartered WGs. Since test suites will be used to make conformance claims, a different life cycle than that of the WG must be foreseen.