Proposed split of Testing Checkpoint 4.6

Wording as of today:
Checkpoint 4.6. Ensure the framework allows for specification
versioning and errata levels [Priority 2]
Requirement from the Process guidelines.

The checkpoint number may change due to a proposed divison of
Guideline 4 into two parts. I'll ignore that here.

Proposed new checkpoints:
========= NEW STUFF ===============
Checkpoint 4.6. Ensure the framework allows for specification
versioning [Priority 2]
It must be possible to extract those tests which pertain to a
designated version of the specification under test.

Checkpoint 4.7. Ensure the framework allows for specification
errata for each version [Priority 2]
It must be possible to extract those tests which pertain to a
designated version of the specification plus all errata issued
against that version up to a designated date.

Checkpoint 4.8. Ensure that the framework allows extraction of the
test suite as it existed on past dates [Priority ?]
It must be possible to re-run a test using the test suite as it
existed as of a designated date in the past. This checkpoint applies
only to officially published snapshots of the test suite, because
conformance claims should be made by using officially published
versions.
(Editors' note: this may be folded into 4.12)

Checkpoint 4.9. Ensure that each test case may be marked so that
the framework will know the versions and errata levels to which
it applies. [Priority ?]
Checkpoints 4.6 and 4.7 anticipate that the test lab will specify
a specification version and errata level (date, meaning to include
all errata published up to that date) when testing a product. This
checkpoint requires maintenance of data for each test case to
allow filtering of tests. The marking scheme SHOULD be capable of
spanning a range of versions to avoid redundant tests that could
become unsynchronized. The marking scheme SHOULD be capable of
designating that a test is to be excluded as of a certain version
of errata level, to allow tests for older versions to be managed
within the same suite.
(Editors' note: if deprecation gets its own checkpoint, as one of
the Dimensions of Variability, it may provide some complications
here. For example, a test case that is mainstream for Version 3 may
be subject to selective filtering in Version 4 if it's deprecated
in 4 and support for deprecated features is optional or variable
in some way.)
========= NEW STUFF ===============

Notice that the WG needs to vote on priorities for 4.8 and 4.9.
.................David Marston

Received on Wednesday, 3 July 2002 11:56:18 UTC