Testcase Metadata
Editor's draft: Last edited $Date: $
Contents
- Introduction
- Metadata elements
- Notes
- Issues
1. Introduction
This document defines a minimal set of metadata elements that can usefully
be applied to test cases that are intended for publication within a conformance
test suite. Defining and providing metadata has proved helpful during the test
development process and also to those who run the tests to verify the conformance
of their implementations.
2. Metadata elements
Identifier
Title
| Name |
Title |
| Description |
The name by which the test case is formally known |
| Rationale |
It may be advantageous to provide a more human-oriented alternative to
the Identifier |
| Required? |
No |
| Syntax |
Text |
| Example |
The CSS Test
Suite uses the term Description for human readable names that
more properly match this Title element. (for example, CSS 2.1
Test Suite: Co>). |
| See Also |
Dublin Core: title
Testcase Metadata: Identifier |
| Comments |
This element is optional because the Identifier will often be
sufficient. Note also that in very large test suites it may be impractical
to define unambiguous Titles. |
Description
| Name |
Description |
| Description |
A representation in words of the nature and characteristics of the test
case. |
| Rationale |
Provides additional context for the test case. |
| Required? |
No |
| Syntax |
Free-form (hypertext) |
| Example |
From WebCGM (@@need link@@): "Test basic CGM-to-HTML link, from a
Application Structure with linkuri APS Attribute within the CGM, to a whole
HTML document, with no behaviors or fragments associated with the link." |
| See Also |
Dublin Core:
description
Testcase Metadata: Purpose |
| Comments |
@@Explain how this differs from Purpose@@ |
Purpose
Status
| Name |
Status |
| Description |
One of an enumerated list of values that can be used to track the state
of a test at a given time. |
| Rationale |
It can be helpful to track the status of a test case during the development
process. |
| Required? |
No |
| Syntax |
Constrained choice from an enumerated list |
| Example |
One of: unconfirmed, new, assigned, pending, accepted, rejected, holding
(see Conformance
Test Process For WCAG 2.0). |
| See Also |
TestFAQ: Contributions |
| Comments |
This metadata element may also be useful after test suite publication,
when it could be used to record the state of challenges to the validity
of the test case. |
SpecRef
| Name |
SpecRef |
| Description |
Identification of the portion of the specification tested by this test
case. |
| Rationale |
Traceability back to the specification. |
| Required? |
Yes. |
| Syntax |
Freeform hypertext or (ideally) an XPath expression. |
| Example |
|
| See Also |
TestFAQ: Coverage |
| Comments |
The more specific this reference is, the better. (Simply pointing to the
beginning of a large sub-section of the spec is unhelpful; identifying the
exact string containing the requirement to be tested is ideal.) |
Preconditions
| Name |
Preconditions |
| Description |
Conditions that must be met before this test is executed. |
| Rationale |
Any such conditions must be understood anc met before the test case can
be successfully executed. |
| Required? |
No |
| Syntax |
Freeform (hypertext). |
|
Example
|
It might be necessary that a network connection be available or a server
process be running before the test case is executed. |
| See Also |
Testcase Metadata: Inputs |
| Comments |
|
| Name |
Inputs |
| Description |
Parameters or data that are needed for the test execution. |
| Rationale |
Must be understood and supplied for test execution. |
| Required? |
No |
| Syntax |
Implementation dependent |
| Example |
|
| See Also |
Testcase Metadata: Preconditions |
| Comments |
|
ExpectedResults
| Name |
ExpectedResults |
| Description |
The results that a conformant implementation is expected to produce
when this test case is executed.
|
| Rationale |
If these are not defined, it will be impossible to determine whether the
test case passed or failed. |
| Required? |
Yes |
| Syntax |
Implementaiton dependant |
| Example |
TestFAQ: Outcome |
| See Also |
Test-cases within the CSS Test Suite embed their expected results within
the test's output. See this
example. |
| Comments |
|
Version
| Name |
Version |
| Description |
An identifier which allows one to distinguish between different revisions
of test case. |
| Rationale |
Test cases often evolve over time, and it's important to maintain a history
and to be able to identify and distinguish between different revisions.. |
| Required? |
Yes |
| Syntax |
Implementation dependent
|
| Example |
@@need an example of versioning when applied to a test case, not to an
entire test suite@@ |
| See Also |
TestFAQ: Maintenance |
| Comments |
This will often be generated by a source-code control system such as CVS. |
Contributor
| Name |
Contributor |
| Description |
The individual or organization that contributed this test case. |
| Rationale |
It may be necessary to contact the contributor to ask for information
about the test case, or to request a update. |
| Required? |
Yes |
| Syntax |
Freeform |
| Example |
The XML Conformance Test
Suite encodes the contributor in the directory structure used to store
the test cases. |
| See Also |
Dublin Core:
contributor |
| Comments |
|
Rights
| Name |
Rights |
| Description |
Information about rights held in and over the test case. |
| Rationale |
Publishers and users of the test case need to know where the legal rights
lie. |
| Required? |
Yes |
| Syntax |
Ask your lawyer. |
| Example |
|
| See Also |
Dublin Core: rights |
| Comments |
This will often be simply a pointer to a copyright notice contained within
the source code. |
Grouping
| Name |
Grouping |
| Description |
A mechanism for classifying test cases into groups. |
| Rationale |
To enable the selection of subsets of test cases that share certain characteristics. |
| Required? |
No |
| Syntax |
Implementation dependent. Pssibilities include naming conventions and
enumerated lists. |
| Example |
Tests may be classified as interactive or automated, positive or negative,
voice or dtmf. |
| See Also |
|
| Comments |
An important use of grouping techniques is to classify tests that belong
to particular profiles, modules, or other discretionary groupings (see
Variability
in Specifications for a detailed discussion of these issues). The
SVG test suite, for
example, uses naming conventions to distinguish between test cases targeted
at different profiles (full, basic, or tiny).
|
SeeAlso
| Name |
SeeAlso |
| Description |
A list of references to relevant materials. |
| Rationale |
Can help to clarify the intent or usefulness of the test case. |
| Required? |
No |
| Syntax |
Freeform (hypertext) |
| Example |
A pointer to an item in an issue-tracking system such as Bugzilla
or to a mailing list thread in which the justification for this test case
is discussed. |
| See Also |
Dublin Core:
relation |
| Comments |
|
3. Notes
Notes: metadata about metadata: dates, language
4. Issues
| ID |
Issue |
State |
Owner |
Resolution |
| 1 |
How to distinguish between Purpose & Description?
(Do we really need both? In practice, people won't make the distinction.
The description should state the purpose?) |
Open |
Patrick |
|
| 2 |
Need to make clear distinction between Preconditions and Input |
Open |
Patrick |
|
| 3 |
Is there any value to the Required field, since this document
is non-normative? |
Open |
Patrick |
|
| 4 |
Are the specific references to TestFAQ (to individual questions) really
useful? Wouldn't it be sufficient to link to the entire document in a References
section? |
Open |
Patrick |
|
| 5 |
Is there any value to qualifying Freeform with hypertext?
(Isn't it obvious that hypertext should be used in a freeform field where
appropriate?) |
Open |
Patrick |
|
| 6 |
Need to emphasize the importance of grouping for selection according to
DOV |
Open |
Patrick |
|
| |
|
|
|
|
| |
|
|
|
|
| |
|
|
|
|
Last modified $Date: $ by $Author: $
