Re: Conformance Disclaimer -- comments please

I actually think we should keep #2 in.  To me it means that even if you 
only fail a subset of the test suite that is targeted for a specific (set 
of) requirement(s), you still fail (i.e., you do not conform).  Thus, any 
failure of a specific feature means you fail (like a pass/fail grade in a 
class).  It complements #1 which says, in contrast, that passing everything 
does not guarantee conformance (I would take out the word "full" from #1.)

At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote:

>QAWG,
>
>I have pretty much finished the final WG-review version of OpsGL, for Last 
>Call resolutions.  Since the last WG-discussion draft (20030120), I have 
>been mostly tweaking the wording and adding more "Rationale" sections.
>
>But I have discovered one last clarification issue, and I need your 
>feedback.  OpsGL CP6.4, Conformance Disclaimer.
>
>Two sections follow:  the complete 20021220 text of CP6.4; and, my current 
>(partial) revision.  Following the two sections is my question(s).
>
>### 20021220 text ###
>Checkpoint 6.4. Provide a conformance verification disclaimer with the 
>test materials.  [Priority 1]
>
>To fulfill this checkpoint, the Working Group MUST provide a prominent 
>disclaimer about the use of the test materials for conformance 
>verification of implementations.
>
>Discussion. Although tests suites may be used for conformance 
>verification, the Working Group must make users aware that:
>
>    1. passing all of the tests does not guarantee full conformance of an 
> implementation to the specification
>    2. failing the test suite means failing tests for the specific feature 
> they target
>
>An example of a conformance disclaimer may be found in the Conformance 
>chapter of this specification.
>### end ###
>
>### current editing progress ###
>Checkpoint 6.4. Provide a conformance verification disclaimer with the 
>test materials.
>
>Conformance requirements: the Working Group MUST provide a prominent 
>disclaimer about the use of the test materials for conformance 
>verification of implementations.
>
>Rationale.  It is common to draw unwarranted conclusions about conformance 
>to the specification from test suite results.  A conformance disclaimer 
>clarifies the relationship between test suite results and conformance.
>
>(@@unchanged from here on@@)Discussion. Although tests suites may be used 
>for conformance verification, the Working Group must make users aware that:
>
>1. passing all of the tests does not guarantee full conformance of an 
>implementation to the specification
>2. failing the test suite means failing tests for the specific features 
>they target.
>
>An example of a conformance disclaimer may be found in the Conformance 
>chapter of this specification.
>### end ###
>
>Questions:
>-----
>
>a.) What does #2 mean? (It is hard to parse.)
>
>It seems like "they" really refers to the test suite.  I.e., is the 
>intended statement something like, "Failing the test suite means failing 
>(some?) tests for specific features targeted by the test suite."?
>
>If so... then so what?  What does that say about conformance?
>
>b.)  Are we trying to say (disclaim) something like, "If you fail some 
>tests and therefore fail the test suite, don't try to draw any conclusions 
>beyond the scope of the specific features targeted by the test 
>suite."?  And is that true?!
>
>c.)  Isn't it true that failing one specific-feature test for a MUST 
>requirement of the specification means that the implementation does not 
>conform to the specification?  Maybe that does not sound like 
>"disclaimer", but if it is true, why aren't we saying that?  (Is it too 
>obvious?)
>
>Maybe I'm missing the point altogether, and #a-c are way off.  In any 
>case, if this is clear to you, please comment.
>
>-Lofton.
>
>

****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Wednesday, 29 January 2003 18:02:22 UTC