Re: Outline for TestGL-Lite

Comment embedded....

At 12:37 PM 3/12/04 +0000, Andrew Thackrah wrote:

>Here's a thought about Principle #4
>
>If the TS is being used to make conformance claims then the impact on 
>conformance of making any change to the TS must be considered.
>
>Company A uses TS version 1 and makes a conformance claim.
>Then, tests are changed, maybe bugs fixed etc and TS version 2 is released
>Company B uses TS version 2 and makes a conformance claim.
>
>Are the claims equivalent?
>Also does TS version 2 immediately supercede version 1 on publication or 
>does v. 1 continue to be valid (either indefinitely or for a period of 
>'overlap' time)?
>
>In some Open Group certification programs we deal with this by having a 
>specific policy concerning the nature of the claim, requirement to fix 
>identified bugs in a certain period, overlap period of maintenance 
>releases of test suites etc. This stuff is important because there are 
>contracts and legal requirements involved.
>In a less formal testing process the issues are still present and probably 
>SHOULD be considered.
>
>
>This implies:
>     * The meaning/value of a conformance claim may change as the test 
> suite evolves.
>       o Instructions on making conformance claims should address this

That's a good point.

In SpecLite, "how to make a Conformance Claim" is one of the dozen+ items 
that survived.  Since our new approach is to clearly and completely develop 
each of a much-reduced number of topics -- with explanation, examples, 
markup, tools, templates, etc -- would this cover your point?  Or are you 
suggesting that something about this belongs in TestTL also or instead?

-Lofton.



>-Andrew
>
>Patrick Curran wrote:
>>Here's a brief outline for a "lite" version of TestGL. Dimitris: I added 
>>a couple of principles (addressing usability and usefulness) but we still 
>>have only four.
>>Comments?
>
>>     Principle #4: Test suites must evolve over time
>>     * To meet needs of changing specs (revisions)
>>     * To improve quality/coverage
>>     * To fix bugs found during development, testing, or use of the test
>>       suite
>>This implies:
>>     * Must plan for multiple releases of the test suite
>>     * Each release must be versioned (people must know what version
>>       they're using)
>>     * Must be able to accept andrespond to bug reports
>>           o Against individual tests, the docs, the harness, etc.
>>           o Possible responses to bug reports:
>>                 + Patch existing test suite by
>>                       # excluding broken tests from the test suite
>>                       # creating and distributing alternate tests
>>                       # updating docs, harness, framework, etc.
>>                 + Release an updated version of the entire test suite
>>                 + Re-releasing the entire test-suite, even if the
>>                   changes are minor, might be the simplest and least
>>                   confusing way to release updates
>

Received on Monday, 15 March 2004 09:57:39 UTC