The following table defines a range of possible commitments. Each level is based on a combination of the level of testability of the specification and the extent of associated test materials. Specific minimal conformance requirements are given for each level.
Level | testability of specification | extent of test materials |
---|---|---|
1. | Summary. Working Group plans regular specification reviews for
testability, following Specification
Guidelines (SpecGL)
Requirements: MUST commit to completing a SpecGL Implemetation Conformance statement (ICS) at each document milestone of (@@...each /TR/ publication? or, LC, CR, PR, Rec?@@) |
|
2. | Summary. In addition to the previous level, Working Group provides
a set of test assertions, not necessarily complete, before beginning
development of a test materials.
Requirements: MUST commit that a set of test assertions for each of the WG's specifications will be publicly available at the time that test materials development commences, and it SHOULD be located on the WG "/Test/" page (see CP4.4). |
Summary. Working Group ensures that a test suite (@@test
materials?@@) for the specification, not necessarily complete and
thorough, will exist by the time the specification becomes
Recommendation.
Requirements: MUST commit that each of the WG's specifications will have an associated test suite (@@test materials?@@) before it becomes Recommendation. |
3. | Summary. In addition to the previous level, a Working Group will
provide numerous use cases and examples in the body of the
specification.
Requirements: MUST commit that specification(s) will satisfy SpecGL CP1.2, 1.3, 1.4. |
Summary. In addition to the previous level, Working Group ensures
that any test suite (@@TM?@@) provided by the WG is reviewed by the
Working Group or externally and the review process meets the criteria
established in this document.
Requirements: MUST commit that test suites (@@TMs?@@) meet the review requirements of the checkpoints in this guidelines specification, specifically CP5.4 (@@..tbd, are there other CPs here?@@) |
4. | Same as previous. | Summary. In addition to the previous level, Working Group will
establish and maintain a test cases contribution and review process
(@@Ed note..."and will produce and review the test suite, not
necessarily complete and thorough,"...these words repeat L2 and L3,
and should be deleted@@) before the specification becomes
Recommendation.
Requirements: MUST commit to establish and maintain contribution and review process (@@link to CPs@@) before Rec. |
5. | Summary. In addition to the previous level, Working Group will
derive the list of testable assertions as an addendum to
specification by the time it becomes Recommendation.
Requirements: MUST commit that specification(s) will satisfy SpecGL CP14.1 (@@and CP14.2?@@) before the specification becomes Recommendation. |
Same as previous. |
6. | Same as previous. | Summary. In addition to the previous level, a Working Group
insists on a complete test suite before a specification becomes
Recommendation.
Requirements: MUST commit to a complete test suite (@@TMs?@@) for each spec before it becomes Rec. [Ed note. I think there are problems with "complete": 1.) undefined; 2.) impossible to attain.] |
7. | Same as previous. | Summary. In addition to the previous level, Working Group plans to
maintain the test suite even after specification becomes
Recommendation. Working Group plans to support versions and errata in
the test suite and review appeals.
Requirements: MUST commit to maintenance of the TS (@@TM?@@) for each spec after it becomes Rec; MUST commit to version/errata support for the TS (@@TM?@@) of each spec after it becomes Rec. |