Checkpoint 1.1 Table

To all,

As promised in Friday's telcon, I reviewed the table associated with 
Checkpoint 1.1.  With the current checkpoint, this is what we require: a WG 
commitment that a test suite (with no quality control) will exist prior to 
Recommendation and that the WG aims to have numerous normative use cases in 
the body of the Recommendation.

These are the problems I see with the current checkpoint and table:

1) We've asked the WG to commit to a test suite (in Level 2), but we don't 
ask for a commitment to review the test suite until level 4.  Since we're 
the Q/A group, I think any requirement to produce something (a test suite), 
with no provisions to ensure adequate quality, is not appropriate.

2) Although we've asked for commitment to the existence of a test suite in 
level 2, we do not ask for test assertions (as am addendum) until level 
6.  Thus, we could have a test suite developed before the issuance of test 

3) Level 3 asks for the WG to aim to have numerous normative use cases in 
the Rec.  The word "numerous" is vague and unverifiable.  Additionally, I'm 
not sure what is meant by use cases at all, in this context.

I propose the following solutions:

1) Add to level 2, in the testability of the specification column, "Working 
Group commits to develop a set of test assertions, not necessarily 
complete, before beginning development of a test suite."

2) Under testability of the specification, level 3, quantify "numerous 
normative use cases" (i.e., one use case per ???).  In addition, explain 
what is meant by "use cases", in this context.  If neither of the above can 
be done, I suggest removing this requirement.

3) Change Checkpoint 1.1 to "plan to have at least a commitment to Q/A 
level four."  This way, we can guarantee that the test suite will be at 
least of minimal quality (it will have been reviewed and there will be a 
process in place for establishing and maintaining test case 
contributions).  Without review and a plan for test cases, all we've asked 
for in our level 2 requirement is something called a test suite, which 
could be completely garbage.  A bad test suite is much worse than no test 
suite at all.

4) Add to level 5, in the testability of the specification column, "In 
addition to the commitment for the previous level, insists on a complete 
set of test assertions before completing test suite."

In summary, I think the table and checkpoint need some work.  The above 
thoughts are my preliminary ideas, but there may be better ways to address 
the problems.


Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174

Received on Monday, 11 March 2002 11:42:17 UTC