- From: Lofton Henderson <lofton@rockynet.com>
- Date: Wed, 29 Jan 2003 18:49:55 -0700
- To: www-qa-wg@w3.org
Mark, Kirill, David, Lynne -- Thanks for your comments, all. I can see the virtue of several of the arguments (like below) against removal of #2. But also ... as I read it, we have 4 different ideas for exactly what to do with it (#2). Clearly, we have something here that is more than editorial (which is where I started out, "What do we mean, what are we trying to say?"). I have generated and posted the substantively-final text (for Monday resolutions), and I'll send pointers tomorrow, Thursday (gotta' run now). I'd like to propose that #2 is removed for the LC version, and (Mark or Kirill or David) raise an issue during Last Call review. Then we can properly consider and argue alternatives (including precise proposed text). Does this sound like an acceptable way forward? (I see no real alternative way forward, unless someone thinks that he/she can propose exact revised wording, that everyone will agree to by email before Monday.) -Lofton. At 05:53 PM 1/29/03 -0500, Mark Skall wrote: >I actually think we should keep #2 in. To me it means that even if you >only fail a subset of the test suite that is targeted for a specific (set >of) requirement(s), you still fail (i.e., you do not conform). Thus, any >failure of a specific feature means you fail (like a pass/fail grade in a >class). It complements #1 which says, in contrast, that passing >everything does not guarantee conformance (I would take out the word >"full" from #1.) > >At 07:32 AM 1/29/2003 -0700, Lofton Henderson wrote: > >>QAWG, >> >>I have pretty much finished the final WG-review version of OpsGL, for >>Last Call resolutions. Since the last WG-discussion draft (20030120), I >>have been mostly tweaking the wording and adding more "Rationale" sections. >> >>But I have discovered one last clarification issue, and I need your >>feedback. OpsGL CP6.4, Conformance Disclaimer. >> >>Two sections follow: the complete 20021220 text of CP6.4; and, my >>current (partial) revision. Following the two sections is my question(s). >> >>### 20021220 text ### >>Checkpoint 6.4. Provide a conformance verification disclaimer with the >>test materials. [Priority 1] >> >>To fulfill this checkpoint, the Working Group MUST provide a prominent >>disclaimer about the use of the test materials for conformance >>verification of implementations. >> >>Discussion. Although tests suites may be used for conformance >>verification, the Working Group must make users aware that: >> >> 1. passing all of the tests does not guarantee full conformance of an >> implementation to the specification >> 2. failing the test suite means failing tests for the specific >> feature they target >> >>An example of a conformance disclaimer may be found in the Conformance >>chapter of this specification. >>### end ### >> >>### current editing progress ### >>Checkpoint 6.4. Provide a conformance verification disclaimer with the >>test materials. >> >>Conformance requirements: the Working Group MUST provide a prominent >>disclaimer about the use of the test materials for conformance >>verification of implementations. >> >>Rationale. It is common to draw unwarranted conclusions about >>conformance to the specification from test suite results. A conformance >>disclaimer clarifies the relationship between test suite results and >>conformance. >> >>(@@unchanged from here on@@)Discussion. Although tests suites may be used >>for conformance verification, the Working Group must make users aware that: >> >>1. passing all of the tests does not guarantee full conformance of an >>implementation to the specification >>2. failing the test suite means failing tests for the specific features >>they target. >> >>An example of a conformance disclaimer may be found in the Conformance >>chapter of this specification. >>### end ### >> >>Questions: >>----- >> >>a.) What does #2 mean? (It is hard to parse.) >> >>It seems like "they" really refers to the test suite. I.e., is the >>intended statement something like, "Failing the test suite means failing >>(some?) tests for specific features targeted by the test suite."? >> >>If so... then so what? What does that say about conformance? >> >>b.) Are we trying to say (disclaim) something like, "If you fail some >>tests and therefore fail the test suite, don't try to draw any >>conclusions beyond the scope of the specific features targeted by the >>test suite."? And is that true?! >> >>c.) Isn't it true that failing one specific-feature test for a MUST >>requirement of the specification means that the implementation does not >>conform to the specification? Maybe that does not sound like >>"disclaimer", but if it is true, why aren't we saying that? (Is it too >>obvious?) >> >>Maybe I'm missing the point altogether, and #a-c are way off. In any >>case, if this is clear to you, please comment. >> >>-Lofton. >> > >**************************************************************** >Mark Skall >Chief, Software Diagnostics and Conformance Testing Division >Information Technology Laboratory >National Institute of Standards and Technology (NIST) >100 Bureau Drive, Stop 8970 >Gaithersburg, MD 20899-8970 > >Voice: 301-975-3262 >Fax: 301-590-9174 >Email: skall@nist.gov >**************************************************************** >
Received on Wednesday, 29 January 2003 20:47:16 UTC