Conformance Disclaimer -- comments please

QAWG,

I have pretty much finished the final WG-review version of OpsGL, for Last 
Call resolutions.  Since the last WG-discussion draft (20030120), I have 
been mostly tweaking the wording and adding more "Rationale" sections.

But I have discovered one last clarification issue, and I need your 
feedback.  OpsGL CP6.4, Conformance Disclaimer.

Two sections follow:  the complete 20021220 text of CP6.4; and, my current 
(partial) revision.  Following the two sections is my question(s).

### 20021220 text ###
Checkpoint 6.4. Provide a conformance verification disclaimer with the test 
materials.  [Priority 1]

To fulfill this checkpoint, the Working Group MUST provide a prominent 
disclaimer about the use of the test materials for conformance verification 
of implementations.

Discussion. Although tests suites may be used for conformance verification, 
the Working Group must make users aware that:

    1. passing all of the tests does not guarantee full conformance of an 
implementation to the specification
    2. failing the test suite means failing tests for the specific feature 
they target

An example of a conformance disclaimer may be found in the Conformance 
chapter of this specification.
### end ###

### current editing progress ###
Checkpoint 6.4. Provide a conformance verification disclaimer with the test 
materials.

Conformance requirements: the Working Group MUST provide a prominent 
disclaimer about the use of the test materials for conformance verification 
of implementations.

Rationale.  It is common to draw unwarranted conclusions about conformance 
to the specification from test suite results.  A conformance disclaimer 
clarifies the relationship between test suite results and conformance.

(@@unchanged from here on@@)Discussion. Although tests suites may be used 
for conformance verification, the Working Group must make users aware that:

1. passing all of the tests does not guarantee full conformance of an 
implementation to the specification
2. failing the test suite means failing tests for the specific features 
they target.

An example of a conformance disclaimer may be found in the Conformance 
chapter of this specification.
### end ###

Questions:
-----

a.) What does #2 mean? (It is hard to parse.)

It seems like "they" really refers to the test suite.  I.e., is the 
intended statement something like, "Failing the test suite means failing 
(some?) tests for specific features targeted by the test suite."?

If so... then so what?  What does that say about conformance?

b.)  Are we trying to say (disclaim) something like, "If you fail some 
tests and therefore fail the test suite, don't try to draw any conclusions 
beyond the scope of the specific features targeted by the test 
suite."?  And is that true?!

c.)  Isn't it true that failing one specific-feature test for a MUST 
requirement of the specification means that the implementation does not 
conform to the specification?  Maybe that does not sound like "disclaimer", 
but if it is true, why aren't we saying that?  (Is it too obvious?)

Maybe I'm missing the point altogether, and #a-c are way off.  In any case, 
if this is clear to you, please comment.

-Lofton.

Received on Wednesday, 29 January 2003 09:37:06 UTC