LC comment for SpecGL : 'Should provide a disclaimer template'

Here is a last call comment from David Marston (David_Marston@us.ibm.com) 
on QA Framework : Specifications Guidelines (and Examples and Techniques)
received by the LC form system.

Submitted on behalf of: N/A
Comment type: Substantive
The comment applies to: "11.3 Provide a conformance disclaimer."
Comment title: Should provide a disclaimer template

Comment:
When I visited the joint XSL/XQ WG session (March 6 in Cambridge),
there was quite a bit of discussion about the claims and disclaimers
suggested in Guideline 11 of SpecGL. WG members attending wanted to
have something more concrete to start with, like a "boilerplate"
paragraph or two. In particular, we talked about number of test cases
passed as a bad metric for conformance, because it implies that each
case has equal weight. The current checkpoints don't call out this
practice as warranting discouragement.

Proposed resolution : 
Include in Ck 11.3, and probably 11.2 as well, a template that editors
can paste into their specs. The template should include sentences
addressing particular good (11.2) or bad (11.3) practices that might
occur, allowing editors to remove irrelevant sentences.

I think that for 11.2, you already know that you intend a template
resembling: This [class of product] was tested for conformance to
[name of spec] version [N.nn], [Nth] Edition, dated [date], and all
errata issued through [date], ... more specs cited same way ....
The testing occurred on [date] using [test suite identifier] and
[name of product and version] was found to conform at [level] level
except for [enumeration of failing tests]. A full report of the
parameter settings for the test harness and all results is posted at
[URL] for open, public review.

For 11.3, the template could look like this: While the test suite
provides [hundreds, thousands] of test cases, not all cases should be
considered to carry equal weight. A product that passes all the tests
may still not conform in some untested area. The W3C hereby states
that claims of passage of a certain number of test cases or a certain
percentage of the test cases, but not all, are invalid as relative
measurements of conformance or worthiness, and that the only valid
data that can be derived from such a result is that the product
being tested does not pass all the tests. [Optional: More tests may
be added to the suite in the future, and existing tests may be
changed when errata are discovered. Failing some test cases cannot be
interpreted as failing to conform without corroboration.] [Optional:
The test suite can be tailored to suit permissible variability in
product behavior. The W3C encourages implementers to provide
information in their Implementation Conformance Statement that will
lead to accurate configuration of the test suite, but holds the test
lab responsible for obtaining the information and tailoring the suite
accordingly, or else reporting which pieces of information were
undetermined and indicating that some test failures may in fact be
due to configuration problems.]
]]

-- 
This comment was submitted through the lastCall form system,
designed by Martin Duerst and Adapted for the QAWG by Olivier Thereaux.

Received on Wednesday, 19 March 2003 12:44:38 UTC