More checkpoints for GL 3 of TestGL?

Reflecting on TestGL, I see checkpoint 3.3 (analyze test coverage) as 
different from the others, since it ties into the specs. You analyze 
coverage by comparing what you have against what you want to have, and the 
test assertions (coming from the specs) are an essential tool for defining 
what you want to have. One could argue that this checkpoint fits better in 
Guideline 5, which has checkpoints about the quality of the suite. One 
could even argue that it goes under Guideline 1, if you set a coverage 
policy first, then compile the list of holes that need filling. Guideline 
3 claims to be about managing the materials systematically, but this 
checkpoint is as much about the holes as about the tests that fill them.

With that background, Patrick's closing question about whether there are 
other checkpoints needed in GL 3 is really question about other 
checkpoints related to managing the list of needs (holes). Should GL 3 be 
about both test cases and needs? If so, I'd like CP 3.4 to be reworded 
from "bug-tracking system" to "issue-tracking system" so it's clear that 
there is a range of situations that can occur. I don't think we want to 
hint, let alone mandate, that there must be a separate tracking system for 
test case needs. A WG could have a separate system, driven by test 
assertions, if they wish. "Needed" might also be a status code in a system 
that tracks the status of all the cases. Tracking the Needed state is 
especially important for those WGs that intend to write their own suite as 
a WG task, rather than accept submissions.

Regarding issues tracking, notice that checkpoint 7.2 (solicit feedback on 
the TM) would also feed into such a system. Checkpoint 1.3 and most of the 
checkpoints under GL 5 could raise issues.

Errata can change the status of test cases (from Needs Review to Approved, 
from Approved to Needs Review, and possibly from Rejected to Approved) but 
can also drive the creation of new test case needs.

Another possible area for additional checkpoints involves the repair of a 
test case that is buggy buts addresses a need. As far as I can tell, the 
existing checkpoints are likely to address all the facets that should be 
codified in TestGL.
.................David Marston

Received on Monday, 22 September 2003 13:27:19 UTC