[ISSUE 8] Technique - Provide Wording for Conformance Claim

 From the minutes of QA WG Teleconf 2004-09-27
http://www.w3.org/mid/415AFE26.3080604@sun.com

AI-20040927-4 KD    Mail discussion of how to reconcile issue  
#8    2003-10-04 

[ISSUE 8] Technique - Provide Wording for Conformance Claim
 http://www.w3.org/QA/WG/2004/09/qaspec-issues#no-008


In Spec GL WD Edition 2004-08-30
http://www.w3.org/TR/2004/WD-qaframe-spec-20040830/#specify- 
conformance#conformance-claim-tech

Technique
 1.   Define your conformance model and identify the conformance labels  
you are using.
 2.   Give all information identifying uniquely the specification:  
name, versioning, date.
 3.   Provide a template of the conformance claims, which includes  
placeholders for the following:
 ◦   the specification name,
 ◦  the specification version,
 ◦  the satisfied conformance label,
 ◦  information about the subject which is claiming conformance,
 ◦  the date of the claim.
 4.   The specification should require that conformance claims contain  
at least this information.

Proposal of David Marston
http://lists.w3.org/Archives/Public/www-qa-wg/2004Aug/att-0087/00-part


Proposed Resolution:

Technique

1. Define your conformance model and identify the conformance labels  
you are using.
2. Give all information identifying uniquely the specification: name,  
versioning, date.
3. Provide a template of the conformance claims, which includes  
placeholders for the following:
    3.1. The version number of the Recommendation...
    3.2. The date of the errata document...
    3.3. Which modules are implemented... [assuming XFoo has modules]
 [3.4 Removed]
    3.5. The nature of any extensions implemented...
    3.6. The version identifier of the W3C conformance test suite  
applied...
 [3.7 Removed]
 3.8 which discretionary items implemented

4. The specification should require that conformance claims contain at  
least this information.
6. Disclaim, on behalf of the W3C, any public claim of conformance that  
does not include certain enumerated details of the claim.
7. Describe how a software buyer can use the pro forma and the  
requirements for a conformance claim in their own RFQ verbiage.



What is RFQ in the point 7?

Added 3.8
Added 6 and 7.

Don't forget we don't want to have a too heavy text. Sometimes I  
believe the text you have given is very informative and very valuable.  
So two proposals for it.

* An appendix in SpecGL
* Or a wiki page

-> both giving details about conformance claims and their implications  
in the market.


I have removed:

 """3.4. Which choices were made on all the
 implementation-dependent and implementation-defined
 items in Appendix D, preferably by using the Pro
 Forma outlined in Appendix H."""
-> the language is a bit terse and might be difficult to understand for  
the spec developpers. I'm not sure we loose a lot without it.

 """3.7. Sufficient details of how the tests
 were applied to allow re-running of the test
 suite in an independent test environment.
 These details must show that the test suite
 was applied in accordance with the instructions
 provided at [reference]..."""
-> This is a very good one but not for the conformance claim, but for  
the Test Suite itself. The test suite should make sure that everything  
will give unique results or if they are dependencies on the platform to  
give information how to handle them. In a sense, the test suite, if  
necessary, should define profiles and the conformance claim in this  
case will refer to a profile.

 """8. Disclaim any conformance statement that
 says that less than 100% of the applicable test
 cases passed, therefore the product is "partially
 conformant", "X% conformant", "X% passing", "conformant
 in all but [feature X]", etc. Use the discretionary
 items, other DoV, and the pro forma to determine the
 set of "applicable tests" for the product."""
-> It's not a technique, it can be in the prose of why care for example  
or it could be the topic of a wiki page with all the conformance claims  
problems. But the prose of the why care/what does it mean, could  
include something saying that a product can't be half conformant and  
that only the conformance claim can be used in the terms defined by the  
conformance claim itself.

 """9. Disavow the notion that the WG-sanctioned
 test suite is complete and that passing 100%
 of applicable tests means that the product
 is (fully) conformant."""
-> same. More for the prose of What does that mean, explaining it and  
for all the following points.

10. Disavow the notion that individual test cases in the suite all have  
equal weight. (Example: the product that fails 20 test cases could be  
worse than the product that fails 100 cases, depending on which cases  
and the buyer's needs.)
11. Provide instructions on proper administration of the WG-sanctioned  
test suite and require that any test reports state what steps were  
taken where the instructions allow site-specific adaptation. (Example:  
if actual and expected output are compared in a site-specific way, the  
source code for the comparitor routine(s) must be published.)
12. Provide wording that is usable by a vendor applying the tests to  
their own product, an assumed-hostile vendor applying the tests to a  
competing product, and an independent test labs testing several  
products.


-- 
Karl Dubost - http://www.w3.org/People/karl/
W3C Conformance Manager
*** Be Strict To Be Cool ***