Re: Revised Guidelines and Checkpoints for TestGL

Le lun 18/08/2003 à 02:41, Patrick Curran a écrit :
> Checkpoint 1.2. Identify the specification(s) to be tested. [Priority 1]
> Conformance requirements: All specifications referenced by the specification under test must be identified. 
> The extent to which these specifications can be assumed to be tested, or must be tested by the conformance test suite for this specification, must be stated.

Avoid passive form for CR: the object that must conform to the
requirement should be the subject, ie:
"The test suite (test documentation?) must identify all specifications
referenced by the specfication under test."...

(this applies to many other CP, too).

> Checkpoint 2.2. Tag assertions with essential metadata [Priority 1]
> Conformance requirements: The Working Group must define the required set of metadata that will be associated with test assertions. This must include at least the following data:

Don't talk about the WG; a test suite may be developped by another
entity than the WG, and more importantly, a WG doens't have to conform
to TestGL, only Test Materials have to. Possible reformulation:
"the Test Suite documentation must define the required set of
metadata..."

This applies to several other CP as well.

> Checkpoint 2.3. Tag assertions with additional useful metadata [Priority 2]
> Conformance requirements: Additional metadata may be associated with test assertions. This should include at least the following data:
    
Avoid may and should as much as possible (this applies to several other
CP as well).

> Checkpoint 6.1 Tests should report their status in a consistent manner [Priority 1]
> Conformance requirements: Tests must report their execution status in a consistent manner. At a minumum, tests should report whether they passed, failed, or whether the results were inconclusive.

What about speaking about the test suite (test harness?) rather than
tests? (id for 6.2). 'Tests' seems too fuzzy IMO. Something like:
"The test harness must report the execution status of the tests in a
consistent manner". 

> [ED NOTE: do we need to add more states?]

FWIW, EARL defines 5 possible states:
"""
Instances of ValidityLevel
    * cannotTell
    * fail
    * notApplicable
    * notTested
    * pass
"""
http://www.w3.org/TR/2002/WD-EARL10-20021206/#validity-instances

> Guideline 7. Plan for conformance testing
> [ED NOTE: shouldn't this be in OpsGL rather than here?]

definitely.

Dom
-- 
Dominique Hazaël-Massieux - http://www.w3.org/People/Dom/
W3C/ERCIM
mailto:dom@w3.org