Re: Question on Developing Test Assertions

I think assertions should be derived from normative sources.
So it's a matter of identifying what information is normative and
recording this somewhere in the spec (the conformance section?). If there
is any useful additional information then it should be declared as
normative for conformance purposes.

This is only relevant for conformance testing though. I think it's ok
to have an assertion set that is derived from any sources that the
test suite designer thinks will be useful (e.g. for in house development).
Since each assertion will (should) be tagged to a source document, any assertions in the
set that are derived from normative sources can be used for
conformance tests, the others can not.

I suppose the bootstrapping problem is inevitable when developing a spec
and tests together - but that's why we encourage it - to flush out
the problems earlier rather than later. It should mean for more work in
the short term, less in the long (in theory..)

-Andrew

On Tue, 3 Feb 2004, Mark Skall wrote:

>
> Now that we've finished going through the test assertions I generated for
> SpecGL, it seems there is still one unanswered question: How are we going
> to develop test assertions (and modify the existing SpecGL assertions) in
> the future?
>
> There are at least 2 possible scenarios:
>
> 1) Develop the test assertions the same way I developed the SpecGL ones
> (i.e., use the conformance requirements PLUS additional info from
> Rationale, Discussion, Examples, etc.)  I used this additional info because
> I felt more info was needed to fully define (and test) the conformance
> requirements.
>
> 2) Use ONLY the conformance requirements - no more no less - to generate
> the assertions.  In my mind, this is the ideal, and preferred,
> approach.  The conformance requirements section SHOULD (MUST?) be complete
> and self-contained.  Using any additional info really means we're adding
> non-normative requirements to the test assertions.  However, in order to
> use this approach, the conformance requirements have to be scrutinized to
> ensure completeness.
>
> Perhaps, there is a 2-tiered approach:  Use the additional info (scenario
> 1) the first time the assertions are generated.  The main intent would be
> to flush out problems/inconsistencies in the guideline.  Then use ONLY the
> conformance requirements (scenario 2) to re-do the assertions for the final
> version.
>
> Any thoughts?
>
> Mark

Received on Tuesday, 3 February 2004 17:29:04 UTC