Draft Proposed Answer to Ian Hickson: Conformance is not a yes/no proposition (wrt filling an ICS)

Original comment (issue 1041 [1])

http://lists.w3.org/Archives/Public/www-qa/2005Jan/0006.html
<http://lists.w3.org/Archives/Public/www-qa/2005Jan/0006.html> 

 

Thank you for your comment, which the QA Working Group has accepted.  We
have reworded the affected sections to clarify the meaning and use of the
Implementation Conformance Statement (ICS), including explaining the meaning
of checking Yes/No.  The affected sections now read: 

 

Good Practice 04, Provide an Implementation Conformance Statement Pro Forma

"What does it mean? An Implementation Conformance Statement (ICS) provides
information about an implementation to a specification, by presenting in a
uniform manner the implemented capabilities (e.g., functions, features) and
options as well as limitations of the implementation. An ICS pro forma
typically takes the form of a blank questionnaire or checklist for an
implementation.  It provides the implementer a way to indicate the features
implemented. Think of it as an inventory of what has been implemented.  Note
that a completed ICS does not indicate conformance of the implementation.
Hence, answering "yes" to indicate a capability is supported does not mean
that the capability has been tested."

 

This Good Practice suggests that the specification itself include an ICS pro
forma.  Providing this pro forma makes it conducive to completing and helps
to ensure consistency among completed ICS. 

 

Why care? An ICS pro forma provides a concise summary of a specification,
i.e., the capabilities and options defined in the specification as well as
any defined subdivisions (e.g., profiles, modules) and conformance
designations. The ICS provided with the specification is blank, waiting for
the implementer to complete.  This blank ICS provides implementers and users
a quick overview of features defined in the specification. A completed ICS
not only provides information on what has been implemented (mandatory and
optional features), but can also be used to document the presence of
extensions or any specializations that have been made.  A completed ICS
provides information useful to facilitate the selection of applicable tests
for the particular implementation.  However, that is not all.  Although the
ICS content is independent of testing, associating it with conformance tests
makes it an essential piece in the reporting of conformance results (see 

techniques in Good Practice 05)."

 

Changes were also made to related Good Practice 05 [2]

 

Good Practice 05: Require an ICS as part of valid conformance claims.

"Why Care:  Providing a completed ICS with the conformance claim might help
customers and users to determine quickly the implemented capabilities as
well as easily verify the level of support for individual requirements of
the specifications.  Combining the ICS with a conformance test suite, can
strengthen the claim.  Specifically, the ICS augmented with links to
conformance tests, provides a very nice way to indicate not only what has
been implemented, but also, what has been implemented correctly (i.e.,
conforms to the specification).

Technique: 

Augment the ICS by providing links to the test suite, such that each feature
has associated with it a test (or set of tests).  Explain what it means to
check Yes or No.  Specifically, does Yes/No indicate that the implementation
has the relevant feature and passes the applicable tests or does Yes/No only
indicate that the feature is implemented. In the latter case, add an
additional column, to indicate the result of executing the tests.  To avoid
confusion as to the role of ICS, we recommend adding an additional column. "

 

[1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=1041
<http://www.w3.org/Bugs/Public/show_bug.cgi?id=1041> 

[2] http://www.w3.org/TR/2005/WD-qaframe-spec-20050428/#ics-gp
<http://www.w3.org/TR/2005/WD-qaframe-spec-20050428/#ics-gp>  

[3] http://www.w3.org/TR/2005/WD-qaframe-spec-20050428/#ics-claim-gp
<http://www.w3.org/TR/2005/WD-qaframe-spec-20050428/#ics-claim-gp>