W3C home > Mailing lists > Public > www-qa-wg@w3.org > January 2003

Re: Conformance Disclaimer -- comments please

From: Patrick Curran <Patrick.Curran@sun.com>
Date: Wed, 29 Jan 2003 18:30:06 -0800
Message-ID: <3E388E2E.9090107@sun.com>
To: Kirill Gavrylyuk <kirillg@microsoft.com>
CC: www-qa-wg@w3.org

OK - now I understand what the intent of #2 is. However, I have to say 
that I don't think this intent is clearly expressed by the current wording.

Thanks for clarifying...

Kirill Gavrylyuk wrote:

>Hi Patrick,
>Welcome to the group!
>  
>
>>b) If you fail one test, you are definitely non-conformant
>>    
>>
>With the #2 it gets complicated if the conformance clause assumes
>multiple level of conformance or discretionary. Then you may have an
>implementation failing some tests, but still conformant to the spec (may
>be with the lower level).
>
>I tend to vote to leave it as is unless we can make the wording to
>respect the cases of multi-level conformance, other dimensions of
>variability...
>
>  
>
>>-----Original Message-----
>>From: Patrick Curran [mailto:Patrick.Curran@sun.com]
>>Sent: Wednesday, January 29, 2003 3:21 PM
>>To: www-qa-wg@w3.org
>>Subject: Re: Conformance Disclaimer -- comments please
>>
>>
>>First, let me introduce myself. I'm Patrick Curran from Sun
>>Microsystems, and I have just joined the group as Sun's
>>    
>>
>representative.
>  
>
>>I'm the manager of the team that develops the JCK (the conformance
>>    
>>
>test
>  
>
>>suite for Java 2, Standard Edition), and I also develop conformance
>>    
>>
>test
>  
>
>>suites for various XML technologies in the Java platform. I will
>>    
>>
>attend
>  
>
>>Monday's meeting, and introduce myself more fully then.
>>
>>As for this issue, I think we need to say two things:
>>
>>a) Just because you passed all the tests, you can't claim to be
>>conformant - all you can say is that you passed all the tests
>>b) If you fail one test, you are definitely non-conformant
>>
>>In other words, a conformance test suite can never prove conformance,
>>    
>>
>it
>  
>
>>can only disprove it.
>>
>>The wording below for item 1 seems fine to me, but I don't think that
>>the wording in #2 expresses what we need to say.
>>
>>Mark Skall wrote:
>>
>>    
>>
>>>I actually think we should keep #2 in.  To me it means that even if
>>>you only fail a subset of the test suite that is targeted for a
>>>specific (set of) requirement(s), you still fail (i.e., you do not
>>>conform).  Thus, any failure of a specific feature means you fail
>>>(like a pass/fail grade in a class).  It complements #1 which says,
>>>      
>>>
>in
>  
>
>>>contrast, that passing everything does not guarantee conformance (I
>>>would take out the word "full" from #1.)
>>>
>>>At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote:
>>>
>>>      
>>>
>>>>QAWG,
>>>>
>>>>I have pretty much finished the final WG-review version of OpsGL,
>>>>        
>>>>
>for
>  
>
>>>>Last Call resolutions.  Since the last WG-discussion draft
>>>>(20030120), I have been mostly tweaking the wording and adding more
>>>>"Rationale" sections.
>>>>
>>>>But I have discovered one last clarification issue, and I need your
>>>>feedback.  OpsGL CP6.4, Conformance Disclaimer.
>>>>
>>>>Two sections follow:  the complete 20021220 text of CP6.4; and, my
>>>>current (partial) revision.  Following the two sections is my
>>>>question(s).
>>>>
>>>>### 20021220 text ###
>>>>Checkpoint 6.4. Provide a conformance verification disclaimer with
>>>>the test materials.  [Priority 1]
>>>>
>>>>To fulfill this checkpoint, the Working Group MUST provide a
>>>>prominent disclaimer about the use of the test materials for
>>>>conformance verification of implementations.
>>>>
>>>>Discussion. Although tests suites may be used for conformance
>>>>verification, the Working Group must make users aware that:
>>>>
>>>>   1. passing all of the tests does not guarantee full conformance
>>>>        
>>>>
>of
>  
>
>>>>an implementation to the specification
>>>>   2. failing the test suite means failing tests for the specific
>>>>feature they target
>>>>
>>>>An example of a conformance disclaimer may be found in the
>>>>Conformance chapter of this specification.
>>>>### end ###
>>>>
>>>>### current editing progress ###
>>>>Checkpoint 6.4. Provide a conformance verification disclaimer with
>>>>the test materials.
>>>>
>>>>Conformance requirements: the Working Group MUST provide a
>>>>        
>>>>
>prominent
>  
>
>>>>disclaimer about the use of the test materials for conformance
>>>>verification of implementations.
>>>>
>>>>Rationale.  It is common to draw unwarranted conclusions about
>>>>conformance to the specification from test suite results.  A
>>>>conformance disclaimer clarifies the relationship between test
>>>>        
>>>>
>suite
>  
>
>>>>results and conformance.
>>>>
>>>>(@@unchanged from here on@@)Discussion. Although tests suites may
>>>>        
>>>>
>be
>  
>
>>>>used for conformance verification, the Working Group must make
>>>>        
>>>>
>users
>  
>
>>>>aware that:
>>>>
>>>>1. passing all of the tests does not guarantee full conformance of
>>>>        
>>>>
>an
>  
>
>>>>implementation to the specification
>>>>2. failing the test suite means failing tests for the specific
>>>>features they target.
>>>>
>>>>An example of a conformance disclaimer may be found in the
>>>>Conformance chapter of this specification.
>>>>### end ###
>>>>
>>>>Questions:
>>>>-----
>>>>
>>>>a.) What does #2 mean? (It is hard to parse.)
>>>>
>>>>It seems like "they" really refers to the test suite.  I.e., is the
>>>>intended statement something like, "Failing the test suite means
>>>>failing (some?) tests for specific features targeted by the test
>>>>suite."?
>>>>
>>>>If so... then so what?  What does that say about conformance?
>>>>
>>>>b.)  Are we trying to say (disclaim) something like, "If you fail
>>>>some tests and therefore fail the test suite, don't try to draw any
>>>>conclusions beyond the scope of the specific features targeted by
>>>>        
>>>>
>the
>  
>
>>>>test suite."?  And is that true?!
>>>>
>>>>c.)  Isn't it true that failing one specific-feature test for a
>>>>        
>>>>
>MUST
>  
>
>>>>requirement of the specification means that the implementation does
>>>>not conform to the specification?  Maybe that does not sound like
>>>>"disclaimer", but if it is true, why aren't we saying that?  (Is it
>>>>too obvious?)
>>>>
>>>>Maybe I'm missing the point altogether, and #a-c are way off.  In
>>>>        
>>>>
>any
>  
>
>>>>case, if this is clear to you, please comment.
>>>>
>>>>-Lofton.
>>>>
>>>>
>>>>        
>>>>
>>>****************************************************************
>>>Mark Skall
>>>Chief, Software Diagnostics and Conformance Testing Division
>>>Information Technology Laboratory
>>>National Institute of Standards and Technology (NIST)
>>>100 Bureau Drive, Stop 8970
>>>Gaithersburg, MD 20899-8970
>>>
>>>Voice: 301-975-3262
>>>Fax:   301-590-9174
>>>Email: skall@nist.gov
>>>****************************************************************
>>>
>>>      
>>>
>>    
>>
>
>  
>
Received on Wednesday, 29 January 2003 21:30:33 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:12 GMT