Re: Mark's comments on Checkpoint 1.1 Table

At 12:10 PM 3/11/02 -0500, David Marston/Cambridge/IBM wrote:
> >1) We've asked the WG to commit to a test suite (in Level 2), but we
> >don't ask for a commitment to review the test suite until level 4.
>
>I would hope that the WGs are already sufficiently motivated to issue
>only tests that they "accept" through their internal procedures.


I've found from my long and painful experience to never assume anything, 
especially when writing a standard or guideline - make sure every 
requirement is in writing.


> >2) Although we've asked for commitment to the existence of a test suite
> >in level 2, we do not ask for test assertions (as am addendum) until
> >level 6.
>
>When I originated the set of levels, I knew that I was taking notions
>from parallel tracks and trying to make a sensible sequence. In my QA
>identity, I want test assertions to be in the table at as low a level
>(numerically) as possible. When I think about what has to be "sold to"
>the substantive WGs, however, the idea of a set of test assertions
>seems (to me) to require quite a lot of convincing. Specifically, it
>has a large impact on the information content of the Recommendation
>document, and in that sense more impact on their work than the
>development of any test suite other than a complete one.


Again, if we believe it is a requirement, I believe, we need to state 
it.  Part of our mission is education.  Test assertions are a mandatory and 
necessary condition for the development of a good test suite.  There are 
only 2 possibilities wrt to impact on their work: 1) It will have a large 
impact on their work (like you say) because they were developing a test 
suite without doing test assertions.  In this case, it needs to have this 
impact because the test suite would have probably been garbage or 2) It 
will have no impact on their work because, in the course of developing the 
test suite, they had already developed test assertions.  The bottom line is 
this:  test assertions are a necessary prerequisite to a test suite and 
thus good quality implementations and interoperable solutions.  Why sugar 
coat it?

> >3) Level 3 asks for the WG to aim to have numerous normative use cases
> >in the Rec.  The word "numerous" is vague and unverifiable.
>
>And the way you would set that number might be as a percentage of the
>number of test assertions. Along with Mark's idea that the hoped-for
>lowest level of commitment might move up to Level Four, this pushes
>the topic of test assertions to the forefront.
>
>What reaction will you get if you (in effect) insist that all Recs
>ought to have a complete set of test assertions? If that position is
>too far advanced beyond current practice, how do you compromise?


You don't compromise - you educate.  We're not asking for a complete set of 
test assertions (in my proposal) until Level 5 (when we ask for a complete 
test suite). This is not Priority 1 so we're NOT insisting that all recs 
have a complete set of test assertions.  However, by the time we have a 
complete test suite (level 5), we need to have already had complete test 
assertions.

We can't have a complete test suite (worth anything) without complete test 
assertions.  In any case, we need test assertions BEFORE the corresponding 
part of the test suite that has been developed.



>Lessen insistence to weaker compulsion?
>Sacrifice completeness and ask for them to attempt to be complete?
>Insist on differentiation of testable statements from plain text?
>.................David Marston

****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Monday, 11 March 2002 16:40:12 UTC