TestGl comments

Some comments on the new version of QA Test Guideline Working Draft.


Guideline 4. Provide a test framework

General comment: It seems to me that the concept of  "test harness" was 
replaced by  "test framework".  Test harness is associated with running 
tests, a test framework supports a test suite development by providing 
guidance in terms of the kind of metadata that needs to be provided, 
documentation and use of tests (this is in line with the definition given 
in the "Definitions" section). Basically, this discrepancy is getting me 
confuse specially when I move on to GL4  which seems to be associating the 
concept of test framework specifically with running the tests or testsuite, 
using the definition of GL3 -- no where in this guideline is mentioned the 
other aspects of the test framework as described in the "Definition" section.

Checkpoint 4.1. The test framework definition should be clearly stated in 
this checkpoint. Test framework was defined under Guideline 3 as "a System 
that assists in the running of the tests." The term test framework in 
relation to running tests should be re-visited. The definition given to 
test framework in the definition section does not match this one (in GL3).

Checkpoint 4.3: Comment:  Automation of testing should emphasized the 
ability of automatically execute the test suite as a whole and also allow 
for execution of test cases or specific test cases based on a particular 
criteria (like, association with areas of the specification). I just feel 
that the last sentence needs to be expanded a little bit more.

Checkpoint 5.1: This checkpoint is basically requiring the use of a result 
reporting mechanism or system. This result reporting system that we are 
requiring is a "result-reporting framework". This checkpoint should 
introduce and define the terminology "result reporting framework" prior to 
get into the details in the checkpoints that follows.

Recommendation:

Checkpoint 5.1. Test Materials must provide a results reporting framework.

         Conformance requirements. Test materials must support a method of 
recording the pass/fail state for each test.

Rationale: Providing a results reporting framework allows implementers to 
submit a uniform set of results that will help ease the implementation 
comparison process needed to determine interoperability.  It also provides 
data that supports the process of identifying overall and feature specific 
level of support among implementations.


Cordially,

Sandra

Sandra I. Martinez
National Institute of Standards and Technology
100 Bureau Drive, Stop 8970,
Gaithersburg, Md. 20899

(301) 975-3579
sandra.martinez@nist.gov

Received on Friday, 23 May 2003 09:37:37 UTC