This document defines a set of common guidelines for conformance test
materials for W3C
specifications. This document is one in a family of Framework documents of
the Quality Assurance (QA) Activity, which includes the
other existing or in-progress specifications QA Framework:
Introduction,QA Framework: Operational Guidelines, and
QA Framework: Specification Guidelines.
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. The latest status
of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the second published Working Draft. It is expected that
updated WD versions of this document
will be produced regularly, along with other members of the Framework
documents family. Future progression of this document beyond Working Draft is
possible, but has not yet been determined.
This part of the Framework document family will eventually have an
informative accompanying QA Framework: Test Examples and
Techniques document. It will illustrate ways in which the guidelines
and checkpoints of this document might be satisfied.
The QA Working Group
Patent Disclosure page contains details on known patents related to this
specification, in conformance with W3C policy
requirements.
At this time,
the QAWG (QA Working Group) has
addressed the contents of the document at a high level, agreeing to the concepts
and principles as well the coverage of the guidelines. The QAWG has not, at this
time, addressed and achieved consensus of the priority levels. A future version
of this document will be accompanied by a "Specification Examples &
Techniques" document, which will illustrate the guidelines and checkpoints with
case studies, and explain how to satisfy the checkpoints.
Please send comments to www-qa@w3.org,
the publicly
archived list of the QA Interest
Group [QAIG]. Please note
that any mail sent to this list will be publicly archived
and available, do not send information you wouldn't want to see distributed,
such as private data.
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or obsoleted by other documents at any time. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress".
A list of current W3C Recommendations and other technical documents can be
found at http://www.w3.org/TR/.
- Introduction
- Scope and goals
- Class of product and audience
- Motivation and expected benefits
- Relationship to other specifications
- Understanding and using this document
- Checkpoint
priorities
- Terminology
- Guidelines
- High
level functional analysis of spec to determine strategy of test development.
- Deep analysis the spec and extract what needs
to be tested and how
- Test management system
- Test suite/cases development
- Test execution.
- Provide a framework for results reporting
- Plan
for conformance testing
- Conformance
- Conformance definition
- Conformance disclaimer
- Definitions
- Acknowledgments
- References
- Change History
The scope of this specification is a set of requirements for Test Materials (TM)
that if satisfied, will enhance the useability and clarity of the test materials.
It covers the analysis and coverage of specifications, the prioritization and
management of test cases, test frameworks and result reporting.
The goal is to help W3C Working Groups (WGs) and test material developers in
developing test materials that provide consistent, reproducible results with
in a well defined and clear scope.
The class of product or target this specification is conformance test
materials including conformance test suites, validation tools, conformance
checklists, and any other materials that are used to check or indicate
conformance.
The intended audience for these guidelines is developers of conformance test
materials for W3C specifications, however it is applicable all W3C Working
Group members as well as the testers of implementations.
While it is optimal that the development of test materials begin early
in the process, these guidelines are intended for
newly chartered and existing Working Groups alike. Working Groups who
may already be doing some of these activities should review the document
and incorporate the principles and guidelines into their test materials
as much as possible.
The development of quality test materials helps enhance the quality of
specifications by identifying aspects of the specification that are
ambiguous, contradictory or un-implementable. By helping to improve
the clarity of the specification, the quality of implementations is also
improved.
Quality test materials also help improve the conformance of implementations
by providing methods of checking conformance to well defined criteria in
a consistent way. By providing a consistent way to check the conformance of
implementations, test materials also lead to more interoperable implementations.
This document is part of a family of QA Framework documents designed to
help the WGs improve all aspects of their quality practices by solidifying
and extending current quality practices found within the W3C. The QA
Framework documents are:
- QA Framework: Introduction [QAF-INTRO]
- QA Framework: Operational Guidelines [QAF-OPS]
- QA Framework: Specification Guidelines [QAF-SPEC]
- QA Framework: Test Guidelines (this document)
The QA Framework documents are interrelated and complement each other. For example,
there is a close relationship between producing and maintaining test materials and
the tracking of errata. Hence there is interrelationship between this document and
the Operational Guidelines. The reader is strongly encouraged to be familiar with
the other documents in the family.
The guidelines are intended for all Working Groups as well as developers of conformance
materials for W3C specifications. Not only are the Working Groups the consumer of these
guidelines they are also key contributors. The guidelines capture the experiences, good
practices, activities, and lessons learned of the Working Groups and present them in a
comprehensive, cohesive set of documents for all to use and benefit from. The objective
is to reuse what works rather than reinvent and to foster consistency across the various
Working Group quality activities and deliverables.
This specification applies the WAI (Web Accessibility Initiative) guidelines model to Working Group
quality practices and the development of conformance test materials. See, for example, Web Content
Accessibility Guidelines. Each guideline includes:
- the guideline number,
- the statement of the guideline,
- some discussion of and rationale behind the guideline,
- a collection of one or more checkpoints.
The guidelines in this document begin with the analysis of the specification and the identification
of priorities and goals of the test materials (GL1-2). The guidelines then focus on test case management
and development (GL3-4). They then proceed to cover useability and result reporting (GL5-7).
The checkpoint definitions in each guideline define test material aspects and systems that
need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:
- the checkpoint number,
- the statement of the checkpoint (its "title"),
- the priority of the checkpoint,
- a statement of the conformance requirements of the checkpoint, i.e., how to fulfill the checkpoint,
- optional informative notes, rationale, clarifying examples, and cross references to related guidelines or checkpoints,
- a link to the corresponding section in Test Examples & Techniques.
Each checkpoint is intended to be specific enough so that someone can implement the
checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will
contain at least one, and may contain multiple individual requirements, that use
RFC2119 normative keywords. See the Conformance section for further discussion of
requirements and test assertions.
Two separate appendices to this document [TEST-CHECKLIST] and [TEST-ICS] present
all checkpoints in a tabular form sorted in their original order and sorted by
their priorities, for convenient reference. The latter is an Implementation
Conformance Statement ( ICS) pro-forma for this specification.
Some checkpoints are more critical than others for the timely production
of high-quality, highly usable test materials. Therefore each checkpoint has
a priority level based on the checkpoint's impact on the quality and timing
of the test materials produced by a Working Group.
- [Priority 1]
- Critical/essential. These checkpoints are considered to be basic requirements
for ensuring that test materials are usable, and are produced in time to ensure
the quality of the standard and its implementations. Satisfying these checkpoints
is a basic requirement to ensure quality and interoperability of the standard.
- [Priority 2]
- Important/desirable. Satisfying these checkpoints, in addition to the priority
1 checkpoints, should significantly improve the usability and timeliness of
the test materials, as well as the quality of the standard and its
implementations.
- [Priority 3]
- Useful/beneficial. Satisfying these checkpoints, in addition to the Priority 1
and Priority 2 checkpoints, will further improve the quality, usability,
and timeliness of the test materials and the standard itself.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used as
defined in RFC 2119 [RFC2119]. When used with the
normative RFC2119 meanings, they are all upper case. Occurrences of
these words in lower case comprise normal prose usage, with no normative
implications.
Unusual terms in this framework document are defined when first used, and
most generally useful QA-specific terms will eventually be in the QA Glossary
[QA-GLOSSARY].
Checkpoints:
Conformance requirements: The test suite MUST define it scopes,
goal, and intended purpose.
Rationale: When writing test suites it is critical to understand their primary purpose
and scope. The scope describes the areas covered by the test suite, thereby indicating
the applicability of the test suite, the motivation and objectives, and coverages.
For example, the goals, intended purpose, and coverage of tests for a CR document may be
different than those for Rec. Note, a WG may have multiple test suites for different
parts of a specification (e.g., profiles or modules) or for different applications of
the specification (e.g., bindings, things like CSS in MathML CSS with WCAG etc)
Conformance requirements:
Rationale:
Conformance requirements:
Rationale:
Conformance requirements:
Rationale:
Checkpoints:
Conformance requirements:
Rationale:
Extract assertions/normative language and tag
according to category - Using catagories provided by patrick
In other words rather than having explicit checkpoints for each required,
optional, depreciated, discressionary, under defined, etc asserts,
have a checkpoint that has all asserts or normative language extracted and
then grouped by category. It's the same basic idea but it takes 4 checkpoints into one.
Conformance requirements:
Rationale:
Checkpoints:
Conformance requirements:
Rationale:
Conformance requirements:
Rationale:
Checkpoints:
Conformance requirements:
Rationale:
Checkpoints:
Conformance requirements: The test management system must be designed such that
any tester, given a set of criteria, will use the same set of tests and
the same set of results from those tests.
Rationale: Test cases must be un ambiguous. The test management system must allow
the selection of test cases based on a set of well defined criteria. If the
test management system allows for different testers to determine differing criteria
for the same target or if test cases are not clear and un ambiguous, different
testers may derive different results and different levels of conformance for the
same target implementation.
Conformance requirements: Develop a test harness that
allows for some or all of the tests to be automated. The test harness should
be useable by the widest audience possible.
Rationale: Reasonably complete test suites may contain
a significantly large number of test cases. In order to allow for the widest
possible coverage, automation is helpful. Automation frameworks should not
be designed such that it may be used with a wide number of platforms and
systems.
Conformance requirements: The test management system
must allow for tests to be filtered and sorted based on a well defined set of
criteria. @@ this will relate to what ever Patrick is doing with GL1-2 and
the metadata stuff... @@@. Filtering must support criteria like profiles,
levels and modules, optional and required tests, and associations to
asserts or normative language in the specification.
Rationale: By allowing testers to sort and filter
test cases based on these criteria, testers can quickly identify and
run only the tests associated with the specific target that the tester
is covering. By associating test cases back to the test assertion,
testers can identify what is being tested and what the correct result
should be.
Conformance requirements: The test management
system must allow for the results of run tests to be associated
with the test.
Rationale: By providing a way for testers to
associate the results of a test they become one more criteria by
which tests may be grouped. By identifying tests that pass, testers
do not need to recover the same ground.
Discussion. Providing support for results helps
to fulfill part of the requirement of Guideline 6 as well.
One of the entrance criteria for a Proposed Recommendation (PR) is that each feature
of the technical report has been implemented, preferably by demonstrating two
interoperable implementations of each feature. Providing a standardized result reporting
mechanism helps facilitate these interoperability tests. A second goal of the WG
should be to encourage vendors to report testing results for their products. In order
to do that, a WG needs to provide vendors with the results format, necessary style sheets,
and any other tools needed to facilitate reporting.
Checkpoints:
Conformance requirements: Test Materials must support a method of
recording the pass/fail state for each test.
Rationale: In the case of interoperability test suites required for
verifying two implementations, providing a system for result reporting
allows implementors to submit uniform sets of results which helps ease
the process of comparison and the determination of what features are in
danger of being cut due to their lack of support.
Conformance requirements: The result reporting framework should allow for uses
to export or otherwise generate a self contained report that may be published on the web.
Rationale: In order to encourage vendors to publicly publish their results on the web
it is desireable to make the process of publishing as easy as possible so that the
process is not an impediment to publishing. By producing a unified web page or package
of pages vendors do not need to convert the results from one format to another in order
to publish.
Conformance requirements: The result reporting framework must indicate what tests
have passed and what tests have failed. For failed tests, some indication of the
reason for failure should also be included in or referenced by the report.
Rationale: The whole purpose of a result reporting framework is to be able to determine
the result of each test. If the framework does not indicate if tests are passing or failing
then it is not doing it's primary job. In order to improve an implementation, the implementors
must know what is failing and how in order to fix it. The more details that are provided
about the failure, the easier it is for the implementors to locate and fix the problem.
Conformance requirements: The result reporting framework should support filtering on
the same metadata criteria as the Test Management System.
Rationale: Results should be filterable based on criteria just as test cases should be.
This allows for implementors to only focus on tests and the results of tests that relate
to the particular target criteria of the implementation while being able to filter out
tests and the results for tests that are not applicable to a given implementation.
Checkpoints of this guideline aim to ensure that the Working Group has a
plan for getting interested parties involved in the development and use of
conformance materials.
Checkpoints:
Conformance requirements: Document a plan to engage
vendors of implementations to participate in conformance testing activities.
Rationale: The conformance testing of various implementations
tends to help improve the interoperability of various implementations.
A common practice is to support public discussion group dedicated to the
test suite and organize face-to-face meetings for vendors and other
interested parties.
Conformance requirements: Provide a special space where
information pertaining to test results of their products can be maintained.
Rationale:
It may be possible for the W3C to have a special space where information
pertaining to test results can be given, if not explicitly, then using links
to those pages where the information can be found (in order not to have to
provide disclaimers).
This section defines conformance of Working Group processes and operations
to the requirements of this specification. The requirements of this
specification are detailed in the checkpoints of the preceding "Guidelines" chapter and apply to the Working
Group QA-related documents and deliverables required by this
specification.
This section defines three levels of conformance to this specification:
- Conformance Level "A": all Priority 1 checkpoints are
satisfied,
- Conformance Level "AA": all Priority 1 and Priority 2
checkpoints are satisfied,
- Conformance Level "AAA": all Priority 1, Priority 2,
and Priority 3 checkpoints are satisfied.
A Working Group conforms to the "QA Framework: Test Guidelines" at Level
X (A, AA, or
AAA) if the Working Group meets at least all Conformance
Level X requirements.
To make an assertion about conformance to this document, specify:
- The guidelines title: "QA Framework: Test Guidelines" and its version
number.
- The guidelines URI: http://www.w3.org/TR/2002/qaframe-test/.
- The conformance level satisfied: "Level A", "Level AA", or "Level
AAA"
- The name of the test materials and its version
- The date of the claim.
Example:
The Test Suite for X module, version 2.1 of this Working Group, conforms
to the W3C's 'QA Framework: Test Guidelines' version 1.0, available at
http://www.w3.org/TR/2002/qaframe-test/, Level AA as determined on January
1, 2003.
The checkpoints of this specification present verifiable conformance
requirements about the quality of the test materials developed or adopted by
the Working Group. As with any verifiable test requirements, users should be
aware that:
- Passing all of the requirements to achieve a given conformance level --
A, AA, or AAA -- does not guarantee that the quality of the subject test
materials are well-suited to or will achieve their intended purposes.
- Failing to achieve level A conformance does not mean that the subject
test materials are necessarily deficient to their intended purposes. It
means that the test materials fail one or more checkpoints that
best-practice experience has shown to facilitate and enable successful
development, maintenance and use of the test materials.
This section contains the definitions for all the critical terms used in
the guidelines below. This does not substitute the QA Glossary
[QA-GLOSSARY], but rather focuses on the most
important terms for the Testing guidelines. Some terms in this section
have been borrowed or adapted from other specifications.
- contradictory behaviors
- Two or more different behaviors unconditionally prescribed by a
specification for the same class of implementations under the same
circumstances.
- discretionary choices
- A value or behavior may be chosen from a well-defined enumerated set
of two or more possibilities.
- optional behaviors
- A well-defined feature may be supported or not (if supported, then
the requirements are clear and unambiguous).
- explicitly undefined behaviors
- Specification states that it is open ended and undefined, what set of
values an element or attribute may have, or the behaviors of a product
that implements a feature.
- test assertion
- A set of premises that are known to be true by definition in the
spec.
- test area
- A minimal compound unit in the test suite structure.
- test framework
- A set of utilities, stylesheets and documentation that describe and facilitate
development, documentation and use of the tests.
- results verification
- A common testing practice used to determine if a test passes or fails by
verification of the test result or output against the expected one.
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C),
- Dimitris Dimitriadis (Ontologicon)
- Karl DuBost (W3C),
- Peter Fawcett (RealNetworks)
- Kirill Gavrylyuk (Microsoft),
- Dominique Hazael-Massieux (W3C)
- Lofton Henderson (CGM Open)
- Ian Jacobs (W3C)
- David Marston (Lotus Development Corp)
- Jack Morrison (Sun Microsystems)
- Lynne Rosenthal (NIST)
- Mark Skall (NIST)
- Andrew Thackrah (Open Group)
- Olivier Thereaux (W3C)
- EXTERN-TA
- QA activity email thread about third-party
participation in test materials production, available at
http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html.
- MATRIX
- W3C-wide conformance activity survey covering all the Working Groups,
"The Matrix",
available at http://www.w3.org/QA/TheMatrix.
- PROCESS
- W3C Process
Document, 19 July 2001, available at
http://www.w3.org/Consortium/Process-20010719/.
- TAXONOMY
- QA Activity test
taxonomy, a classification scheme for conformance test materials,
available at http://www.w3.org/QA/Taxonomy.
- QA-GLOSSARY
- A comprehensive glossary of QA terms, maintained by the QA Working
Group. (Initial version under construction.)
- QAIG
- Quality Assurance Interest
Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/IG/.
- QAWG
- Quality Assurance Working
Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/WG/.
- DOM Working Group
TS
- Process
document for DOM Working Group Test suite, available at
http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115.
- REC-TRACK
- Stages and milestones in the W3C
Recommendation Track, per the Process
Document (Process Document is available at
http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
- RFC2119
- Key words for use in
RFCs to Indicate Requirement Levels, March 1997, available at
http://www.ietf.org/rfc/rfc2119.txt.
- SVGTEST
- SVG Working Group's test suite resource
page, which may be found at
http://www.w3.org/Graphics/SVG/Test/.
- WCAG10
- Web Content Accessibility
Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available
at http://www.w3.org/TR/WCAG10/.
- WG-QA-RANGE
- Email
proposal by David Marston, on the QA public mail list, for range of
Working Group commitment levels to conformance test materials
production, available at
http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html.
- XMLTEST
- OASIS XML Conformance TC's XML
test suite resource page, which may be found at
http://www.oasis-open.org/committees/xml-conformance/.
- XSLT-TEST
- OASIS XML Conformance TC's XSLT/Xpath
test suite resource page, which may be found at
http://www.oasis-open.org/committees/xml-conformance/.
- QAF-TEST
- QA Framework: Test Materials Guidelines (not yet published).
- QAF-SPEC
- "QA Framework: Specification
Guidelines", Working Draft companion version to this document,
available at [...].
- XQuery-use-case
- "XML Query Use
Cases
- SOAP-test
- SOAP
Version 1.2 Specification Assertions and Test Collection
- SOAP12-1
- SOAP Version 1.2 Part
1
- SOAP12-2
- SOAP Version 1.2 Part
2
- XSD-TEST
- W3C
XML Schema Test Collection
- XSLT-DISC
- List
of Discretionary items produced by OASIS XSLT/XPath Technical
Committee
- 3-26-2003
Moved the definitions section to it's new location and updated it.
Started work on new introduction using the same format as the other framework documents.
- 3-12-2003
Added new outline to document commented out a number of sections
that need editing.
- 12-05-2002
Edited, improved the Introduction(goals, motivation, document's
structure
Updated the definition of the checkpoint's Priorities
corrected abstract, SOT
Changed the goal of the document and wording of the
checkpoints/guidelines to focus it on testing strategy, moving all the
tips on tactics to be EX-TECH
Added examples to most of the checkpoints
incorporated all the editors comments
Rewrote several checkpoints in the Gd 5 - defined results
verification and reporting as part of the test framework to remove
redundant checkpoints
Rewrote guideline 6 and 7 to focus on strategy of test development
and testing, rather then on tactics.
Updated the conformance section
- 09-12-2002
Expanded introduction, added motivation, etc...
Added examples to the checkpoints in the Gd1,2,3
[MS] Changed the text of many checkpoints to make them verifiable
[DD] First pass on Introduction, added more text to the checkpoints
in the Gd 3-5
- 07-01-2002
Fixed definitions of priorities
Fixed the glitch with the "Test Areas" guideline
Added clarification to Ck 1.1, 1.2, 1.5 (removed "vague"), 1.6
- 06-17-2002
Added short prose to each checkpoint
- 06-12-2002
First draft outline
