W3C home > Mailing lists > Public > www-qa-wg@w3.org > January 2003

Re: Conformance Disclaimer -- comments please

From: Patrick Curran <Patrick.Curran@sun.com>
Date: Wed, 29 Jan 2003 15:21:00 -0800
Message-ID: <3E3861DC.8020406@sun.com>
To: www-qa-wg@w3.org

First, let me introduce myself. I'm Patrick Curran from Sun 
Microsystems, and I have just joined the group as Sun's representative. 
I'm the manager of the team that develops the JCK (the conformance test 
suite for Java 2, Standard Edition), and I also develop conformance test 
suites for various XML technologies in the Java platform. I will attend 
Monday's meeting, and introduce myself more fully then.

As for this issue, I think we need to say two things:

a) Just because you passed all the tests, you can't claim to be 
conformant - all you can say is that you passed all the tests
b) If you fail one test, you are definitely non-conformant

In other words, a conformance test suite can never prove conformance, it 
can only disprove it.

The wording below for item 1 seems fine to me, but I don't think that 
the wording in #2 expresses what we need to say.  

Mark Skall wrote:

>
> I actually think we should keep #2 in.  To me it means that even if 
> you only fail a subset of the test suite that is targeted for a 
> specific (set of) requirement(s), you still fail (i.e., you do not 
> conform).  Thus, any failure of a specific feature means you fail 
> (like a pass/fail grade in a class).  It complements #1 which says, in 
> contrast, that passing everything does not guarantee conformance (I 
> would take out the word "full" from #1.)
>
> At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote:
>
>> QAWG,
>>
>> I have pretty much finished the final WG-review version of OpsGL, for 
>> Last Call resolutions.  Since the last WG-discussion draft 
>> (20030120), I have been mostly tweaking the wording and adding more 
>> "Rationale" sections.
>>
>> But I have discovered one last clarification issue, and I need your 
>> feedback.  OpsGL CP6.4, Conformance Disclaimer.
>>
>> Two sections follow:  the complete 20021220 text of CP6.4; and, my 
>> current (partial) revision.  Following the two sections is my 
>> question(s).
>>
>> ### 20021220 text ###
>> Checkpoint 6.4. Provide a conformance verification disclaimer with 
>> the test materials.  [Priority 1]
>>
>> To fulfill this checkpoint, the Working Group MUST provide a 
>> prominent disclaimer about the use of the test materials for 
>> conformance verification of implementations.
>>
>> Discussion. Although tests suites may be used for conformance 
>> verification, the Working Group must make users aware that:
>>
>>    1. passing all of the tests does not guarantee full conformance of 
>> an implementation to the specification
>>    2. failing the test suite means failing tests for the specific 
>> feature they target
>>
>> An example of a conformance disclaimer may be found in the 
>> Conformance chapter of this specification.
>> ### end ###
>>
>> ### current editing progress ###
>> Checkpoint 6.4. Provide a conformance verification disclaimer with 
>> the test materials.
>>
>> Conformance requirements: the Working Group MUST provide a prominent 
>> disclaimer about the use of the test materials for conformance 
>> verification of implementations.
>>
>> Rationale.  It is common to draw unwarranted conclusions about 
>> conformance to the specification from test suite results.  A 
>> conformance disclaimer clarifies the relationship between test suite 
>> results and conformance.
>>
>> (@@unchanged from here on@@)Discussion. Although tests suites may be 
>> used for conformance verification, the Working Group must make users 
>> aware that:
>>
>> 1. passing all of the tests does not guarantee full conformance of an 
>> implementation to the specification
>> 2. failing the test suite means failing tests for the specific 
>> features they target.
>>
>> An example of a conformance disclaimer may be found in the 
>> Conformance chapter of this specification.
>> ### end ###
>>
>> Questions:
>> -----
>>
>> a.) What does #2 mean? (It is hard to parse.)
>>
>> It seems like "they" really refers to the test suite.  I.e., is the 
>> intended statement something like, "Failing the test suite means 
>> failing (some?) tests for specific features targeted by the test 
>> suite."?
>>
>> If so... then so what?  What does that say about conformance?
>>
>> b.)  Are we trying to say (disclaim) something like, "If you fail 
>> some tests and therefore fail the test suite, don't try to draw any 
>> conclusions beyond the scope of the specific features targeted by the 
>> test suite."?  And is that true?!
>>
>> c.)  Isn't it true that failing one specific-feature test for a MUST 
>> requirement of the specification means that the implementation does 
>> not conform to the specification?  Maybe that does not sound like 
>> "disclaimer", but if it is true, why aren't we saying that?  (Is it 
>> too obvious?)
>>
>> Maybe I'm missing the point altogether, and #a-c are way off.  In any 
>> case, if this is clear to you, please comment.
>>
>> -Lofton.
>>
>>
>
> ****************************************************************
> Mark Skall
> Chief, Software Diagnostics and Conformance Testing Division
> Information Technology Laboratory
> National Institute of Standards and Technology (NIST)
> 100 Bureau Drive, Stop 8970
> Gaithersburg, MD 20899-8970
>
> Voice: 301-975-3262
> Fax:   301-590-9174
> Email: skall@nist.gov
> ****************************************************************
>
Received on Wednesday, 29 January 2003 18:25:56 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:12 GMT