RE: Conformance Disclaimer -- comments please

Hi Patrick,
Welcome to the group!
> b) If you fail one test, you are definitely non-conformant
With the #2 it gets complicated if the conformance clause assumes
multiple level of conformance or discretionary. Then you may have an
implementation failing some tests, but still conformant to the spec (may
be with the lower level).

I tend to vote to leave it as is unless we can make the wording to
respect the cases of multi-level conformance, other dimensions of
variability...

> -----Original Message-----
> From: Patrick Curran [mailto:Patrick.Curran@sun.com]
> Sent: Wednesday, January 29, 2003 3:21 PM
> To: www-qa-wg@w3.org
> Subject: Re: Conformance Disclaimer -- comments please
> 
> 
> First, let me introduce myself. I'm Patrick Curran from Sun
> Microsystems, and I have just joined the group as Sun's
representative.
> I'm the manager of the team that develops the JCK (the conformance
test
> suite for Java 2, Standard Edition), and I also develop conformance
test
> suites for various XML technologies in the Java platform. I will
attend
> Monday's meeting, and introduce myself more fully then.
> 
> As for this issue, I think we need to say two things:
> 
> a) Just because you passed all the tests, you can't claim to be
> conformant - all you can say is that you passed all the tests
> b) If you fail one test, you are definitely non-conformant
> 
> In other words, a conformance test suite can never prove conformance,
it
> can only disprove it.
> 
> The wording below for item 1 seems fine to me, but I don't think that
> the wording in #2 expresses what we need to say.
> 
> Mark Skall wrote:
> 
> >
> > I actually think we should keep #2 in.  To me it means that even if
> > you only fail a subset of the test suite that is targeted for a
> > specific (set of) requirement(s), you still fail (i.e., you do not
> > conform).  Thus, any failure of a specific feature means you fail
> > (like a pass/fail grade in a class).  It complements #1 which says,
in
> > contrast, that passing everything does not guarantee conformance (I
> > would take out the word "full" from #1.)
> >
> > At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote:
> >
> >> QAWG,
> >>
> >> I have pretty much finished the final WG-review version of OpsGL,
for
> >> Last Call resolutions.  Since the last WG-discussion draft
> >> (20030120), I have been mostly tweaking the wording and adding more
> >> "Rationale" sections.
> >>
> >> But I have discovered one last clarification issue, and I need your
> >> feedback.  OpsGL CP6.4, Conformance Disclaimer.
> >>
> >> Two sections follow:  the complete 20021220 text of CP6.4; and, my
> >> current (partial) revision.  Following the two sections is my
> >> question(s).
> >>
> >> ### 20021220 text ###
> >> Checkpoint 6.4. Provide a conformance verification disclaimer with
> >> the test materials.  [Priority 1]
> >>
> >> To fulfill this checkpoint, the Working Group MUST provide a
> >> prominent disclaimer about the use of the test materials for
> >> conformance verification of implementations.
> >>
> >> Discussion. Although tests suites may be used for conformance
> >> verification, the Working Group must make users aware that:
> >>
> >>    1. passing all of the tests does not guarantee full conformance
of
> >> an implementation to the specification
> >>    2. failing the test suite means failing tests for the specific
> >> feature they target
> >>
> >> An example of a conformance disclaimer may be found in the
> >> Conformance chapter of this specification.
> >> ### end ###
> >>
> >> ### current editing progress ###
> >> Checkpoint 6.4. Provide a conformance verification disclaimer with
> >> the test materials.
> >>
> >> Conformance requirements: the Working Group MUST provide a
prominent
> >> disclaimer about the use of the test materials for conformance
> >> verification of implementations.
> >>
> >> Rationale.  It is common to draw unwarranted conclusions about
> >> conformance to the specification from test suite results.  A
> >> conformance disclaimer clarifies the relationship between test
suite
> >> results and conformance.
> >>
> >> (@@unchanged from here on@@)Discussion. Although tests suites may
be
> >> used for conformance verification, the Working Group must make
users
> >> aware that:
> >>
> >> 1. passing all of the tests does not guarantee full conformance of
an
> >> implementation to the specification
> >> 2. failing the test suite means failing tests for the specific
> >> features they target.
> >>
> >> An example of a conformance disclaimer may be found in the
> >> Conformance chapter of this specification.
> >> ### end ###
> >>
> >> Questions:
> >> -----
> >>
> >> a.) What does #2 mean? (It is hard to parse.)
> >>
> >> It seems like "they" really refers to the test suite.  I.e., is the
> >> intended statement something like, "Failing the test suite means
> >> failing (some?) tests for specific features targeted by the test
> >> suite."?
> >>
> >> If so... then so what?  What does that say about conformance?
> >>
> >> b.)  Are we trying to say (disclaim) something like, "If you fail
> >> some tests and therefore fail the test suite, don't try to draw any
> >> conclusions beyond the scope of the specific features targeted by
the
> >> test suite."?  And is that true?!
> >>
> >> c.)  Isn't it true that failing one specific-feature test for a
MUST
> >> requirement of the specification means that the implementation does
> >> not conform to the specification?  Maybe that does not sound like
> >> "disclaimer", but if it is true, why aren't we saying that?  (Is it
> >> too obvious?)
> >>
> >> Maybe I'm missing the point altogether, and #a-c are way off.  In
any
> >> case, if this is clear to you, please comment.
> >>
> >> -Lofton.
> >>
> >>
> >
> > ****************************************************************
> > Mark Skall
> > Chief, Software Diagnostics and Conformance Testing Division
> > Information Technology Laboratory
> > National Institute of Standards and Technology (NIST)
> > 100 Bureau Drive, Stop 8970
> > Gaithersburg, MD 20899-8970
> >
> > Voice: 301-975-3262
> > Fax:   301-590-9174
> > Email: skall@nist.gov
> > ****************************************************************
> >
> 
>

Received on Wednesday, 29 January 2003 18:50:54 UTC