Answer to Ian Hickson: Conformance is not a yes/no proposition (wrt filling an ICS)

Dear Ian,

Thanks for your comments on the Last Call version of the QA Framework:
Specification Guidelines[0] - 22 November 2004

After two weeks from now (on May 18, 2005), the lack of answer will  
be considered as if you had accepted the comment.

Original comment (issue 1041 [1])
http://lists.w3.org/Archives/Public/www-qa/2005Jan/0006.html

Thank you for your comment, which the QA Working Group has accepted.   
We have reworded the affected sections to clarify the meaning and use  
of the Implementation Conformance Statement (ICS), including  
explaining the meaning of checking Yes/No.  The affected sections now  
read:

Good Practice 04, Provide an Implementation Conformance Statement Pro  
Forma
“What does it mean? An Implementation Conformance Statement (ICS)  
provides information about an implementation to a specification, by  
presenting in a uniform manner the implemented capabilities (e.g.,  
functions, features) and options as well as limitations of the  
implementation. An ICS pro forma typically takes the form of a blank  
questionnaire or checklist for an implementation.  It provides the  
implementer a way to indicate the features implemented. Think of it  
as an inventory of what has been implemented.  Note that a completed  
ICS does not indicate conformance of the implementation.  Hence,  
answering "yes" to indicate a capability is supported does not mean  
that the capability has been tested.”

This Good Practice suggests that the specification itself include an  
ICS pro forma.  Providing this pro forma makes it conducive to  
completing and helps to ensure consistency among completed ICS.

Why care? An ICS pro forma provides a concise summary of a  
specification, i.e., the capabilities and options defined in the  
specification as well as any defined subdivisions (e.g., profiles,  
modules) and conformance designations. The ICS provided with the  
specification is blank, waiting for the implementer to complete. This  
blank ICS provides implementers and users a quick overview of  
features defined in the specification. A completed ICS not only  
provides information on what has been implemented (mandatory and  
optional features), but can also be used to document the presence of  
extensions or any specializations that have been made.  A completed  
ICS provides information useful to facilitate the selection of  
applicable tests for the particular implementation.  However, that is  
not all.  Although the ICS content is independent of testing,  
associating it with conformance tests makes it an essential piece in  
the reporting of conformance results (see
techniques in Good Practice 05).”

Changes were also made to related Good Practice 05 [2]

Good Practice 05: Require an ICS as part of valid conformance claims.
“Why Care:  Providing a completed ICS with the conformance claim  
might help customers and users to determine quickly the implemented  
capabilities as well as easily verify the level of support for  
individual requirements of the specifications.  Combining the ICS  
with a conformance test suite, can strengthen the claim.   
Specifically, the ICS augmented with links to conformance tests,  
provides a very nice way to indicate not only what has been  
implemented, but also, what has been implemented correctly (i.e.,  
conforms to the specification).
Technique:
Augment the ICS by providing links to the test suite, such that each  
feature has associated with it a test (or set of tests).  Explain  
what it means to check Yes or No.  Specifically, does Yes/No indicate  
that the implementation has the relevant feature and passes the  
applicable tests or does Yes/No only indicate that the feature is  
implemented. In the latter case, add an additional column, to  
indicate the result of executing the tests.  To avoid confusion as to  
the role of ICS, we recommend adding an additional column. “

[0] http://www.w3.org/TR/2004/WD-qaframe-spec-20041122/
[1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=1041
[2] http://www.w3.org/TR/2005/WD-qaframe-spec-20050428/#ics-gp
[3] http://www.w3.org/TR/2005/WD-qaframe-spec-20050428/#ics-claim-gp

-- 
Karl Dubost
QA Working Group Chair
http://www.w3.org/QA/WG/

Received on Tuesday, 3 May 2005 15:38:00 UTC