W3C home > Mailing lists > Public > www-qa-wg@w3.org > October 2004

[ISSUE 8] Technique - Provide Wording for Conformance Claim

From: Karl Dubost <karl@w3.org>
Date: Thu, 7 Oct 2004 15:43:40 -0400
Message-Id: <307F15BF-1899-11D9-B80C-000A95718F82@w3.org>
To: 'www-qa-wg@w3.org' <www-qa-wg@w3.org>
 From the minutes of QA WG Teleconf 2004-09-27
http://www.w3.org/mid/415AFE26.3080604@sun.com

AI-20040927-4	KD    Mail discussion of how to reconcile issue  
#8				2003-10-04	

[ISSUE 8] Technique - Provide Wording for Conformance Claim
	http://www.w3.org/QA/WG/2004/09/qaspec-issues#no-008


In Spec GL WD Edition 2004-08-30
http://www.w3.org/TR/2004/WD-qaframe-spec-20040830/#specify- 
conformance#conformance-claim-tech

Technique
	1.  	Define your conformance model and identify the conformance labels  
you are using.
	2.  	Give all information identifying uniquely the specification:  
name, versioning, date.
	3.  	Provide a template of the conformance claims, which includes  
placeholders for the following:
	◦ 	 the specification name,
	◦ 	the specification version,
	◦ 	the satisfied conformance label,
	◦ 	information about the subject which is claiming conformance,
	◦ 	the date of the claim.
	4.  	The specification should require that conformance claims contain  
at least this information.

Proposal of David Marston
http://lists.w3.org/Archives/Public/www-qa-wg/2004Aug/att-0087/00-part


Proposed Resolution:

Technique

1. Define your conformance model and identify the conformance labels  
you are using.
2. Give all information identifying uniquely the specification: name,  
versioning, date.
3. Provide a template of the conformance claims, which includes  
placeholders for the following:
    3.1. The version number of the Recommendation...
    3.2. The date of the errata document...
    3.3. Which modules are implemented... [assuming XFoo has modules]
	[3.4 Removed]
    3.5. The nature of any extensions implemented...
    3.6. The version identifier of the W3C conformance test suite  
applied...
	[3.7 Removed]
	3.8 which discretionary items implemented

4. The specification should require that conformance claims contain at  
least this information.
6. Disclaim, on behalf of the W3C, any public claim of conformance that  
does not include certain enumerated details of the claim.
7. Describe how a software buyer can use the pro forma and the  
requirements for a conformance claim in their own RFQ verbiage.



What is RFQ in the point 7?

Added 3.8
Added 6 and 7.

Don't forget we don't want to have a too heavy text. Sometimes I  
believe the text you have given is very informative and very valuable.  
So two proposals for it.

* An appendix in SpecGL
* Or a wiki page

-> both giving details about conformance claims and their implications  
in the market.


I have removed:

	"""3.4. Which choices were made on all the
	implementation-dependent and implementation-defined
	items in Appendix D, preferably by using the Pro
	Forma outlined in Appendix H."""
-> the language is a bit terse and might be difficult to understand for  
the spec developpers. I'm not sure we loose a lot without it.

	"""3.7. Sufficient details of how the tests
	were applied to allow re-running of the test
	suite in an independent test environment.
	These details must show that the test suite
	was applied in accordance with the instructions
	provided at [reference]..."""
-> This is a very good one but not for the conformance claim, but for  
the Test Suite itself. The test suite should make sure that everything  
will give unique results or if they are dependencies on the platform to  
give information how to handle them. In a sense, the test suite, if  
necessary, should define profiles and the conformance claim in this  
case will refer to a profile.

	"""8. Disclaim any conformance statement that
	says that less than 100% of the applicable test
	cases passed, therefore the product is "partially
	conformant", "X% conformant", "X% passing", "conformant
	in all but [feature X]", etc. Use the discretionary
	items, other DoV, and the pro forma to determine the
	set of "applicable tests" for the product."""
-> It's not a technique, it can be in the prose of why care for example  
or it could be the topic of a wiki page with all the conformance claims  
problems. But the prose of the why care/what does it mean, could  
include something saying that a product can't be half conformant and  
that only the conformance claim can be used in the terms defined by the  
conformance claim itself.

	"""9. Disavow the notion that the WG-sanctioned
	test suite is complete and that passing 100%
	of applicable tests means that the product
	is (fully) conformant."""
-> same. More for the prose of What does that mean, explaining it and  
for all the following points.

10. Disavow the notion that individual test cases in the suite all have  
equal weight. (Example: the product that fails 20 test cases could be  
worse than the product that fails 100 cases, depending on which cases  
and the buyer's needs.)
11. Provide instructions on proper administration of the WG-sanctioned  
test suite and require that any test reports state what steps were  
taken where the instructions allow site-specific adaptation. (Example:  
if actual and expected output are compared in a site-specific way, the  
source code for the comparitor routine(s) must be published.)
12. Provide wording that is usable by a vendor applying the tests to  
their own product, an assumed-hostile vendor applying the tests to a  
competing product, and an independent test labs testing several  
products.


-- 
Karl Dubost - http://www.w3.org/People/karl/
W3C Conformance Manager
*** Be Strict To Be Cool ***

Received on Thursday, 7 October 2004 22:19:14 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:18 GMT