RE: Conformance Disclaimer -- comments please

Sounds like a good way to move forward. Could we record the issue right
away? (so that we don't forget)

> -----Original Message-----
> From: Lofton Henderson [mailto:lofton@rockynet.com]
> Sent: Wednesday, January 29, 2003 5:50 PM
> To: www-qa-wg@w3.org
> Subject: Re: Conformance Disclaimer -- comments please
> 
> 
> Mark, Kirill, David, Lynne --
> 
> Thanks for your comments, all.  I can see the virtue of several of the
> arguments (like below) against removal of #2.  But also ... as I read
it,
> we have 4 different ideas for exactly what to do with it (#2).
> 
> Clearly, we have something here that is more than editorial (which is
> where
> I started out, "What do we mean, what are we trying to say?").
> 
> I have generated and posted the substantively-final text (for Monday
> resolutions), and I'll send pointers tomorrow, Thursday (gotta' run
now).
> 
> I'd like to propose that #2 is removed for the LC version, and (Mark
or
> Kirill or David) raise an issue during Last Call review.  Then we can
> properly consider and argue alternatives (including precise proposed
> text).
> 
> Does this sound like an acceptable way forward?
> 
> (I see no real alternative way forward, unless someone thinks that
he/she
> can propose exact revised wording, that everyone will agree to by
email
> before Monday.)
> 
> -Lofton.
> 
> At 05:53 PM 1/29/03 -0500, Mark Skall wrote:
> 
> >I actually think we should keep #2 in.  To me it means that even if
you
> >only fail a subset of the test suite that is targeted for a specific
(set
> >of) requirement(s), you still fail (i.e., you do not conform).  Thus,
any
> >failure of a specific feature means you fail (like a pass/fail grade
in a
> >class).  It complements #1 which says, in contrast, that passing
> >everything does not guarantee conformance (I would take out the word
> >"full" from #1.)
> >
> >At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote:
> >
> >>QAWG,
> >>
> >>I have pretty much finished the final WG-review version of OpsGL,
for
> >>Last Call resolutions.  Since the last WG-discussion draft
(20030120), I
> >>have been mostly tweaking the wording and adding more "Rationale"
> sections.
> >>
> >>But I have discovered one last clarification issue, and I need your
> >>feedback.  OpsGL CP6.4, Conformance Disclaimer.
> >>
> >>Two sections follow:  the complete 20021220 text of CP6.4; and, my
> >>current (partial) revision.  Following the two sections is my
> question(s).
> >>
> >>### 20021220 text ###
> >>Checkpoint 6.4. Provide a conformance verification disclaimer with
the
> >>test materials.  [Priority 1]
> >>
> >>To fulfill this checkpoint, the Working Group MUST provide a
prominent
> >>disclaimer about the use of the test materials for conformance
> >>verification of implementations.
> >>
> >>Discussion. Although tests suites may be used for conformance
> >>verification, the Working Group must make users aware that:
> >>
> >>    1. passing all of the tests does not guarantee full conformance
of
> an
> >> implementation to the specification
> >>    2. failing the test suite means failing tests for the specific
> >> feature they target
> >>
> >>An example of a conformance disclaimer may be found in the
Conformance
> >>chapter of this specification.
> >>### end ###
> >>
> >>### current editing progress ###
> >>Checkpoint 6.4. Provide a conformance verification disclaimer with
the
> >>test materials.
> >>
> >>Conformance requirements: the Working Group MUST provide a prominent
> >>disclaimer about the use of the test materials for conformance
> >>verification of implementations.
> >>
> >>Rationale.  It is common to draw unwarranted conclusions about
> >>conformance to the specification from test suite results.  A
conformance
> >>disclaimer clarifies the relationship between test suite results and
> >>conformance.
> >>
> >>(@@unchanged from here on@@)Discussion. Although tests suites may be
> used
> >>for conformance verification, the Working Group must make users
aware
> that:
> >>
> >>1. passing all of the tests does not guarantee full conformance of
an
> >>implementation to the specification
> >>2. failing the test suite means failing tests for the specific
features
> >>they target.
> >>
> >>An example of a conformance disclaimer may be found in the
Conformance
> >>chapter of this specification.
> >>### end ###
> >>
> >>Questions:
> >>-----
> >>
> >>a.) What does #2 mean? (It is hard to parse.)
> >>
> >>It seems like "they" really refers to the test suite.  I.e., is the
> >>intended statement something like, "Failing the test suite means
failing
> >>(some?) tests for specific features targeted by the test suite."?
> >>
> >>If so... then so what?  What does that say about conformance?
> >>
> >>b.)  Are we trying to say (disclaim) something like, "If you fail
some
> >>tests and therefore fail the test suite, don't try to draw any
> >>conclusions beyond the scope of the specific features targeted by
the
> >>test suite."?  And is that true?!
> >>
> >>c.)  Isn't it true that failing one specific-feature test for a MUST
> >>requirement of the specification means that the implementation does
not
> >>conform to the specification?  Maybe that does not sound like
> >>"disclaimer", but if it is true, why aren't we saying that?  (Is it
too
> >>obvious?)
> >>
> >>Maybe I'm missing the point altogether, and #a-c are way off.  In
any
> >>case, if this is clear to you, please comment.
> >>
> >>-Lofton.
> >>
> >
> >****************************************************************
> >Mark Skall
> >Chief, Software Diagnostics and Conformance Testing Division
> >Information Technology Laboratory
> >National Institute of Standards and Technology (NIST)
> >100 Bureau Drive, Stop 8970
> >Gaithersburg, MD 20899-8970
> >
> >Voice: 301-975-3262
> >Fax:   301-590-9174
> >Email: skall@nist.gov
> >****************************************************************
> >

Received on Wednesday, 29 January 2003 21:35:42 UTC