- From: Patrick Curran <Patrick.Curran@sun.com>
- Date: Wed, 29 Jan 2003 15:21:00 -0800
- To: www-qa-wg@w3.org
First, let me introduce myself. I'm Patrick Curran from Sun Microsystems, and I have just joined the group as Sun's representative. I'm the manager of the team that develops the JCK (the conformance test suite for Java 2, Standard Edition), and I also develop conformance test suites for various XML technologies in the Java platform. I will attend Monday's meeting, and introduce myself more fully then. As for this issue, I think we need to say two things: a) Just because you passed all the tests, you can't claim to be conformant - all you can say is that you passed all the tests b) If you fail one test, you are definitely non-conformant In other words, a conformance test suite can never prove conformance, it can only disprove it. The wording below for item 1 seems fine to me, but I don't think that the wording in #2 expresses what we need to say. Mark Skall wrote: > > I actually think we should keep #2 in. To me it means that even if > you only fail a subset of the test suite that is targeted for a > specific (set of) requirement(s), you still fail (i.e., you do not > conform). Thus, any failure of a specific feature means you fail > (like a pass/fail grade in a class). It complements #1 which says, in > contrast, that passing everything does not guarantee conformance (I > would take out the word "full" from #1.) > > At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote: > >> QAWG, >> >> I have pretty much finished the final WG-review version of OpsGL, for >> Last Call resolutions. Since the last WG-discussion draft >> (20030120), I have been mostly tweaking the wording and adding more >> "Rationale" sections. >> >> But I have discovered one last clarification issue, and I need your >> feedback. OpsGL CP6.4, Conformance Disclaimer. >> >> Two sections follow: the complete 20021220 text of CP6.4; and, my >> current (partial) revision. Following the two sections is my >> question(s). >> >> ### 20021220 text ### >> Checkpoint 6.4. Provide a conformance verification disclaimer with >> the test materials. [Priority 1] >> >> To fulfill this checkpoint, the Working Group MUST provide a >> prominent disclaimer about the use of the test materials for >> conformance verification of implementations. >> >> Discussion. Although tests suites may be used for conformance >> verification, the Working Group must make users aware that: >> >> 1. passing all of the tests does not guarantee full conformance of >> an implementation to the specification >> 2. failing the test suite means failing tests for the specific >> feature they target >> >> An example of a conformance disclaimer may be found in the >> Conformance chapter of this specification. >> ### end ### >> >> ### current editing progress ### >> Checkpoint 6.4. Provide a conformance verification disclaimer with >> the test materials. >> >> Conformance requirements: the Working Group MUST provide a prominent >> disclaimer about the use of the test materials for conformance >> verification of implementations. >> >> Rationale. It is common to draw unwarranted conclusions about >> conformance to the specification from test suite results. A >> conformance disclaimer clarifies the relationship between test suite >> results and conformance. >> >> (@@unchanged from here on@@)Discussion. Although tests suites may be >> used for conformance verification, the Working Group must make users >> aware that: >> >> 1. passing all of the tests does not guarantee full conformance of an >> implementation to the specification >> 2. failing the test suite means failing tests for the specific >> features they target. >> >> An example of a conformance disclaimer may be found in the >> Conformance chapter of this specification. >> ### end ### >> >> Questions: >> ----- >> >> a.) What does #2 mean? (It is hard to parse.) >> >> It seems like "they" really refers to the test suite. I.e., is the >> intended statement something like, "Failing the test suite means >> failing (some?) tests for specific features targeted by the test >> suite."? >> >> If so... then so what? What does that say about conformance? >> >> b.) Are we trying to say (disclaim) something like, "If you fail >> some tests and therefore fail the test suite, don't try to draw any >> conclusions beyond the scope of the specific features targeted by the >> test suite."? And is that true?! >> >> c.) Isn't it true that failing one specific-feature test for a MUST >> requirement of the specification means that the implementation does >> not conform to the specification? Maybe that does not sound like >> "disclaimer", but if it is true, why aren't we saying that? (Is it >> too obvious?) >> >> Maybe I'm missing the point altogether, and #a-c are way off. In any >> case, if this is clear to you, please comment. >> >> -Lofton. >> >> > > **************************************************************** > Mark Skall > Chief, Software Diagnostics and Conformance Testing Division > Information Technology Laboratory > National Institute of Standards and Technology (NIST) > 100 Bureau Drive, Stop 8970 > Gaithersburg, MD 20899-8970 > > Voice: 301-975-3262 > Fax: 301-590-9174 > Email: skall@nist.gov > **************************************************************** >
Received on Wednesday, 29 January 2003 18:25:56 UTC