Test Case Description Language

To the QAWG:

After several years of designing and working with variations of Test
Case Description Languages (TCDLs) implemented in XML, I intend to
write a W3C Note on the subject. I discussed this with Lofton, and he
thinks it looks like a good match for one of the TTF deliverables 
that we enumerated at Boston.

You can see documentation of an early effort in the package that comes
with the OASIS test suite for XSLT/XPath 1.0 conformance:
http://www.oasis-open.org/committees/document.php?document_id=1916&query_string=num_per_wg%3D10%26sort_type%3D%26sort_direction%3D%26search_val%3D%26wg_abbrev%3Dxslt&wg_abbrev=xslt
Since then, the Dimensions of Variability have become more sharply
defined, so the new document would be written to complement SpecGL,
providing motivation for many of the details that SpecGL wants specs
to have.

Below is the latest snapshot of an outline for such a document.
Feedback is welcome.
.................David Marston

The Test Case Description Language (TCDL) is an XML vocabulary for
describing test materials (TMs), intended to be delivered along with
such materials. It allows each test lab using the materials to get
set up to run tests repeatedly. The design presented here is a
flexible one, intended to be adjusted by each Working Group to fit
the classes of product that would be the subject of their specs,
and the Dimensions of Variability (DoV) present in their specs.

1. Benefits that a TCDL would provide (design goals)
  A. Serve as a catalog/manifest/archive of TMs
  B. Maximize reuse potential of test data and possibly tests
  C. Support for automated test runs and compares of results
  D. Support for filtering of tests by spec version and DoV
     (probably at least semi-automated, depending on test regime)
  E. Support creation of parallel directory trees for results
  F. Provide explanatory and traceability material for logfiles
  G. Support ongoing maintenance/improvements/clarifications of TMs

2. Major Features
  A. Identification data for each test case, including citations
  B. Each case has a scenario that controls how it is run, nature of
     inputs needed, nature of output(s) produced
  C. Names of input and reference output files
     (Output names can also be applied to results of test runs)
  D. Data used for filtering out certain tests (where DoV apply)

3. Noteworthy detail features
  A. Spec citations, multiple ones per test, type can vary from TA
     up to a whole section depending on spec layout
  B. Credit to contributors
  C. Strings to be used in logs and reports
  D. Catalog is in XML, permitting many uses via transformation
  E. Can even aid manual processing
  F. Supported versions expressed as a range
  G. Categories and flags for filtering (tests can be designated to
     apply to a certain module or other DoV; can flag operational
     needs such as a network connection)
  H. Naming of cases only mildly constrained (filetype tags)
  I. Namespaced and versioned to provide for future revision

4. Detailed design

5. Tooling built on TCDL
  A. XSLT stylesheets for filtering
  B. XSLT stylesheets to create site-specific automation scripts
  C. XQuery to find TMs meeting certain criteria
  D. etc.

6. Issues and Implications
  A. Good errata design will help
  B. Unreviewed and disputed tests
  C. Gray areas in the spec
  D. "Files" on other media
  E. Testing extendability without testing extensions
  F. Coverage measurement and where to track needs for more cases

The usual appendices would follow.

Received on Thursday, 12 June 2003 12:07:41 UTC