Testcase Metadata

Editor's draft: Last edited $Date: $


  1. Introduction
  2. Metadata elements
  3. Notes
  4. Issues

1. Introduction

This document defines a minimal set of metadata elements that can usefully be applied to test cases that are intended for publication within a conformance test suite. Defining and providing metadata has proved helpful during the test development process and also to those who run the tests to verify the conformance of their implementations.

2. Metadata elements


Name Identifier
Description An unambiguous reference to the test case
Rationale Enables unambiguous reference to the test case
Required? Yes
Syntax Freeform (URI recommended)
Example http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm
See Also Dublin Core: identifier
Testcase Metadata: Title


Name Title
Description The name by which the test case is formally known
Rationale It may be advantageous to provide a more human-oriented alternative to the Identifier
Required? No
Syntax Text
Example The CSS Test Suite uses the term Description for human readable names that more properly match this Title element. (for example, CSS 2.1 Test Suite: Co>).
See Also Dublin Core: title
Testcase Metadata: Identifier
Comments This element is optional because the Identifier will often be sufficient. Note also that in very large test suites it may be impractical to define unambiguous Titles.


Name Description
Description A representation in words of the nature and characteristics of the test case.
Rationale Provides additional context for the test case.
Required? No
Syntax Free-form (hypertext)
Example From WebCGM (@@need link@@): "Test basic CGM-to-HTML link, from a Application Structure with linkuri APS Attribute within the CGM, to a whole HTML document, with no behaviors or fragments associated with the link."
See Also Dublin Core: description
Testcase Metadata: Purpose
Comments @@Explain how this differs from Purpose@@


Name Purpose
Description An explanation of the reason the test case was developed.
Rationale ???
Required? No
Syntax Free-form (hypertext)
See Also Dublin Core: description
Testcase Metadata: Description
Testcase Metadata: SpecRef
Comments @@Explain how this differs from Description@@


Name Status
Description One of an enumerated list of values that can be used to track the state of a test at a given time.
Rationale It can be helpful to track the status of a test case during the development process.
Required? No
Syntax Constrained choice from an enumerated list
Example One of: unconfirmed, new, assigned, pending, accepted, rejected, holding (see Conformance Test Process For WCAG 2.0).
See Also TestFAQ: Contributions
Comments This metadata element may also be useful after test suite publication, when it could be used to record the state of challenges to the validity of the test case.


Name SpecRef
Description Identification of the portion of the specification tested by this test case.
Rationale Traceability back to the specification.
Required? Yes.
Syntax Freeform hypertext or (ideally) an XPath expression.
See Also TestFAQ: Coverage
Comments The more specific this reference is, the better. (Simply pointing to the beginning of a large sub-section of the spec is unhelpful; identifying the exact string containing the requirement to be tested is ideal.)


Name Preconditions
Description Conditions that must be met before this test is executed.
Rationale Any such conditions must be understood anc met before the test case can be successfully executed.
Required? No
Syntax Freeform (hypertext).


It might be necessary that a network connection be available or a server process be running before the test case is executed.
See Also Testcase Metadata: Inputs


Name Inputs
Description Parameters or data that are needed for the test execution.
Rationale Must be understood and supplied for test execution.
Required? No
Syntax Implementation dependent
See Also Testcase Metadata: Preconditions


Name ExpectedResults

The results that a conformant implementation is expected to produce when this test case is executed.

Rationale If these are not defined, it will be impossible to determine whether the test case passed or failed.
Required? Yes
Syntax Implementaiton dependant
Example TestFAQ: Outcome
See Also Test-cases within the CSS Test Suite embed their expected results within the test's output. See this example.


Name Version
Description An identifier which allows one to distinguish between different revisions of test case.
Rationale Test cases often evolve over time, and it's important to maintain a history and to be able to identify and distinguish between different revisions..
Required? Yes

Implementation dependent

Example @@need an example of versioning when applied to a test case, not to an entire test suite@@
See Also TestFAQ: Maintenance
Comments This will often be generated by a source-code control system such as CVS.


Name Contributor
Description The individual or organization that contributed this test case.
Rationale It may be necessary to contact the contributor to ask for information about the test case, or to request a update.
Required? Yes
Syntax Freeform
Example The XML Conformance Test Suite encodes the contributor in the directory structure used to store the test cases.
See Also Dublin Core: contributor


Name Rights
Description Information about rights held in and over the test case.
Rationale Publishers and users of the test case need to know where the legal rights lie.
Required? Yes
Syntax Ask your lawyer.
See Also Dublin Core: rights
Comments This will often be simply a pointer to a copyright notice contained within the source code.


Name Grouping
Description A mechanism for classifying test cases into groups.
Rationale To enable the selection of subsets of test cases that share certain characteristics.
Required? No
Syntax Implementation dependent. Pssibilities include naming conventions and enumerated lists.
Example Tests may be classified as interactive or automated, positive or negative, voice or dtmf.
See Also  

An important use of grouping techniques is to classify tests that belong to particular profiles, modules, or other discretionary groupings (see Variability in Specifications for a detailed discussion of these issues). The SVG test suite, for example, uses naming conventions to distinguish between test cases targeted at different profiles (full, basic, or tiny).


Name SeeAlso
Description A list of references to relevant materials.
Rationale Can help to clarify the intent or usefulness of the test case.
Required? No
Syntax Freeform (hypertext)
Example A pointer to an item in an issue-tracking system such as Bugzilla or to a mailing list thread in which the justification for this test case is discussed.
See Also Dublin Core: relation

3. Notes

Notes: metadata about metadata: dates, language

4. Issues

ID Issue State Owner Resolution
1 How to distinguish between Purpose & Description? (Do we really need both? In practice, people won't make the distinction. The description should state the purpose?) Open Patrick  
2 Need to make clear distinction between Preconditions and Input Open Patrick  
3 Is there any value to the Required field, since this document is non-normative? Open Patrick  
4 Are the specific references to TestFAQ (to individual questions) really useful? Wouldn't it be sufficient to link to the entire document in a References section? Open Patrick  
5 Is there any value to qualifying Freeform with hypertext? (Isn't it obvious that hypertext should be used in a freeform field where appropriate?) Open Patrick  
6 Need to emphasize the importance of grouping for selection according to DOV Open Patrick  


Valid XHTML 1.0!

Last modified $Date: $ by $Author: $

Copyright © 2005 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.