bug #1041 [Re:... Conformance is not a yes/no proposition]

Disclaimer:  I can't find where this was assigned, or where I volunteered 
for it.  So I might be missing some context or some existing QAWG 
resolution.  That notwithstanding, onward....

I think that there is merit in Ian's comment.  Still, I do believe that it 
is a good goal to simplify the answers in an ICS as much as possible -- the 
yes/no/na of GP1.2B Techniques is a good simple goal.  But as he points 
out, it might not always be possible or practical to do so.  Furthermore, 
if one indeed can simplify the answers to yes/no, then it should be clear 
to the reader of the ICS what 'yes' means and what 'no' means.

Here for example is some work I have set up within CGM Open.  At [1] is a 
product directory of WebCGM viewers.  In order to have the product listed, 
each implementation must agree to fill out an ICS.  An example of one such 
is at [2].  (Each viewer product implementor starts with the same 
"pro-forma" -- blank template -- and fills it in).

[1] http://www.cgmopen.org/webcgm/viewers.html
[2] http://www.ematek.de/viewer-proforma.html

I would like to think that [2] satisfies the spirit of what we intend for 
an ICS -- a good high-level view of what functionality the product has 
implemented.  So for example the "checkpoint" 1.2 in the ICS is about 
"Basic linking behaviors -- frames, highlighting, zooming".

At the same time, it is clear what the answer 'yes' means -- the product 
passes a handful of tests in a Basic Effectivity test suite.  This test 
list is in fact linked from the checkpoint number, "1.2".

There is another feature -- instead of "n/a", I have "comments".  "Not 
applicable" might be a comment.  But the real intention is that the 
implementor here can put a link to a text section, which he/she adds below, 
that gives other useful information, qualification, or clarification.  The 
Ematek viewer [2] has not used the feature.  But we have a QAWG example of 
why this is a good idea -- our own SpecGL ICS evaluations of other 
specifications.  Recall how several of us had trouble choosing one of 
'yes', 'no', or 'n/a' for some requirements on some specifications?

In summary:

** as a goal for ICS, simple is good (but maybe not always possible);
** substantiate/define what y/n/na answers mean in the ICS, when these are 
used;
** different specs might need different approaches to ICS content & design;
** 'comments' might be good in addition to or instead of n/a;

I think this encompasses the gist of what Ian was getting at in his below 
comments.  We should take another look at them, and see if he is suggesting 
anything that couldn't be covered with these principles for ICS techniques.

A couple of other comments about GP1.2B:

1.) " An ICS provides a concise view of a specification's conformance 
model."  I'm not sure that this statement is true.  The conformance model 
is not the point of an ICS.

2.) ICS is missing from the Glossary.  Take the CR definition.
http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/guidelines-chapter#Ck-include-ICS

Regards,
-Lofton.

At 02:53 PM 1/19/2005 +0000, Ian Hickson wrote:

>"1.2 Good Practice B" suggests that an ICS form be provided with
>yes/no questions: "1. Create a list, table or form listing all
>features (capabilities) and indicating if it is mandatory or not. 2.
>Provide space for the implementer to check: Yes, No, Not Applicable".
>However, this is unrealistic. For example, take CSS user agents. How
>is an implementor to determine if he has implemented margin collapsing
>correctly? All that can really be said is that the user agent passes a
>certain set of tests. For any even mildly complicated specification it
>will always be possible to show that a user agent is in some way
>non-compliant, it's just a matter of finding a suitable test.
>
>Therefore I would suggest changing this section to instead suggest
>leaving space for the implementor to list the URIs to (publically
>available) tests that the implementor has used to verify
>interoperability and compliance, listing which tests the implementor
>determined passed in the user agent and which failed (if any).
>Specification authors may wish to provide a list of URIs to the tests
>that form part of the specification's formal test suite (as used to
>check for interoperability as per the CR exit criteria), although
>naturally such a test suite can never be complete enough to really be
>used to claim conformance so implementors would be expected to also
>provide links to other tests that they used.
>
>(The existing suggestions could be kept for the rare specs in which a
>test suite is inappropriate, such as the two examples the spec
>currently gives: the QA spec guidelines and the WCAG. However, this
>applies to very few specifications and so should not IMHO be the
>primary suggestion in the document.)
>
>--
>Ian Hickson               U+1047E                )\._.,--....,'``.    fL
>http://ln.hixie.ch/      U+263A                /,   _.. \   _\  ;`._ ,.
>Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Received on Tuesday, 1 March 2005 04:04:19 UTC