- From: by way of the Lastcall Form <David_Marston@us.ibm.com>
- Date: Thu, 20 Mar 2003 02:44 +0900
- To: www-qa@w3.org
Here is a last call comment from David Marston (David_Marston@us.ibm.com) on QA Framework : Specifications Guidelines (and Examples and Techniques) received by the LC form system. Submitted on behalf of: N/A Comment type: Substantive The comment applies to: "11.3 Provide a conformance disclaimer." Comment title: Should provide a disclaimer template Comment: When I visited the joint XSL/XQ WG session (March 6 in Cambridge), there was quite a bit of discussion about the claims and disclaimers suggested in Guideline 11 of SpecGL. WG members attending wanted to have something more concrete to start with, like a "boilerplate" paragraph or two. In particular, we talked about number of test cases passed as a bad metric for conformance, because it implies that each case has equal weight. The current checkpoints don't call out this practice as warranting discouragement. Proposed resolution : Include in Ck 11.3, and probably 11.2 as well, a template that editors can paste into their specs. The template should include sentences addressing particular good (11.2) or bad (11.3) practices that might occur, allowing editors to remove irrelevant sentences. I think that for 11.2, you already know that you intend a template resembling: This [class of product] was tested for conformance to [name of spec] version [N.nn], [Nth] Edition, dated [date], and all errata issued through [date], ... more specs cited same way .... The testing occurred on [date] using [test suite identifier] and [name of product and version] was found to conform at [level] level except for [enumeration of failing tests]. A full report of the parameter settings for the test harness and all results is posted at [URL] for open, public review. For 11.3, the template could look like this: While the test suite provides [hundreds, thousands] of test cases, not all cases should be considered to carry equal weight. A product that passes all the tests may still not conform in some untested area. The W3C hereby states that claims of passage of a certain number of test cases or a certain percentage of the test cases, but not all, are invalid as relative measurements of conformance or worthiness, and that the only valid data that can be derived from such a result is that the product being tested does not pass all the tests. [Optional: More tests may be added to the suite in the future, and existing tests may be changed when errata are discovered. Failing some test cases cannot be interpreted as failing to conform without corroboration.] [Optional: The test suite can be tailored to suit permissible variability in product behavior. The W3C encourages implementers to provide information in their Implementation Conformance Statement that will lead to accurate configuration of the test suite, but holds the test lab responsible for obtaining the information and tailoring the suite accordingly, or else reporting which pieces of information were undetermined and indicating that some test failures may in fact be due to configuration problems.] ]] -- This comment was submitted through the lastCall form system, designed by Martin Duerst and Adapted for the QAWG by Olivier Thereaux.
Received on Wednesday, 19 March 2003 12:44:38 UTC