Re: Question on Developing Test Assertions

As I think I've made clear, I'm a strong believer in identifying test 
assertions directly in the text of the specification. The kind of 
exercise you've just gone through, whereby you take text from the spec 
(from the conformance requirements) and essentially duplicate it 
(sometimes tweaking the language to make it more formal) leads to a 
clumsy document.

However, I understand your concern about needing to pull information 
from places other than the conformance requirements. For perspective, 
here's how we do things in my Java API world:

Test assertions are "identified" from (anywhere) within the spec. As the 
test developers study the spec they work with the spec authors to 
tighten up the language so that genuinely testable assertions can be found.

We don't embed conformance requirements within the spec, so there is no 
"conflict" between these and test assertions. Rather, our conformance 
requirements are contained in the test suite documentation, and they are 
more general than a "QAWG-style" conformance requirement (which for me 
is basically just a test assertion).

Our conformance requirements say things like:

* You must not modify the tests
* Your implementation must pass all of the tests in all of its operating 
modes
* You must implement all of the specification exactly as written even 
where we don't test

I'm not suggesting that we change the style of our framework documents 
now, so I vote for your option #2. However, I feel that the overly 
formal (boolean-logical) language in which you've couched your 
assertions is unnecessary. I think it's possible to write English 
sentences using RFC2119 keywords that are precise enough to stand as 
both conformance requirements and test assertions.

Having said this, I believe that there's great value in the exercise you 
just carried out, since you identified several areas where what we said 
was not really what we meant (for example, "it's OK for your assertion 
list to contain only two entries").

Thanks...



Mark Skall wrote:

>
> Now that we've finished going through the test assertions I generated 
> for SpecGL, it seems there is still one unanswered question: How are 
> we going to develop test assertions (and modify the existing SpecGL 
> assertions) in the future?
>
> There are at least 2 possible scenarios:
>
> 1) Develop the test assertions the same way I developed the SpecGL 
> ones (i.e., use the conformance requirements PLUS additional info from 
> Rationale, Discussion, Examples, etc.)  I used this additional info 
> because I felt more info was needed to fully define (and test) the 
> conformance requirements.
>
> 2) Use ONLY the conformance requirements - no more no less - to 
> generate the assertions.  In my mind, this is the ideal, and 
> preferred, approach.  The conformance requirements section SHOULD 
> (MUST?) be complete and self-contained.  Using any additional info 
> really means we're adding non-normative requirements to the test 
> assertions.  However, in order to use this approach, the conformance 
> requirements have to be scrutinized to ensure completeness.
>
> Perhaps, there is a 2-tiered approach:  Use the additional info 
> (scenario 1) the first time the assertions are generated.  The main 
> intent would be to flush out problems/inconsistencies in the 
> guideline.  Then use ONLY the conformance requirements (scenario 2) to 
> re-do the assertions for the final version.
>
> Any thoughts?
>
> Mark

Received on Tuesday, 3 February 2004 15:18:27 UTC