Re: Outline for TestGL-Lite

Here's a thought about Principle #4

If the TS is being used to make conformance claims then the impact on 
conformance of making any change to the TS must be considered.

Company A uses TS version 1 and makes a conformance claim.
Then, tests are changed, maybe bugs fixed etc and TS version 2 is released
Company B uses TS version 2 and makes a conformance claim.

Are the claims equivalent?
Also does TS version 2 immediately supercede version 1 on publication or 
does v. 1 continue to be valid (either indefinitely or for a period of 
'overlap' time)?

In some Open Group certification programs we deal with this by having a 
specific policy concerning the nature of the claim, requirement to fix 
identified bugs in a certain period, overlap period of maintenance 
releases of test suites etc. This stuff is important because there are 
contracts and legal requirements involved.
In a less formal testing process the issues are still present and 
probably SHOULD be considered.


This implies:
     * The meaning/value of a conformance claim may change as the test 
suite evolves.
       o Instructions on making conformance claims should address this


-Andrew

Patrick Curran wrote:
> Here's a brief outline for a "lite" version of TestGL. Dimitris: I added 
> a couple of principles (addressing usability and usefulness) but we 
> still have only four.
> 
> Comments?
> 

> 
>     Principle #4: Test suites must evolve over time
> 
>     * To meet needs of changing specs (revisions)
>     * To improve quality/coverage
>     * To fix bugs found during development, testing, or use of the test
>       suite
> 
> This implies:
> 
>     * Must plan for multiple releases of the test suite
>     * Each release must be versioned (people must know what version
>       they're using)
>     * Must be able to accept andrespond to bug reports
>           o Against individual tests, the docs, the harness, etc.
>           o Possible responses to bug reports:
>                 + Patch existing test suite by
>                       # excluding broken tests from the test suite
>                       # creating and distributing alternate tests
>                       # updating docs, harness, framework, etc.
>                 + Release an updated version of the entire test suite
>                 + Re-releasing the entire test-suite, even if the
>                   changes are minor, might be the simplest and least
>                   confusing way to release updates
> 
>

Received on Friday, 12 March 2004 07:37:44 UTC