W3C home > Mailing lists > Public > www-qa@w3.org > October 2001

Re: Conformance and Implementations

From: Lofton Henderson <lofton@rockynet.com>
Date: Tue, 09 Oct 2001 09:44:38 -0600
Message-Id: <4.3.2.7.2.20011009081934.050b2920@terminal.rockynet.com>
To: "Karl Dubost" <karl@w3.org>, "Alex Rousskov" <rousskov@measurement-factory.com>
Cc: www-qa@w3.org
Hi,

I have followed the proposal by Karl, about better-quantified and more 
objective conformance claims, and the comments by Alex, and have comments 
about both.

At 07:11 PM 10/8/2001 +0200, Karl Dubost wrote:
>[...]
>I would be interested by your opinions.

Bottom line.  Improving the assertion of formal conformance claims, as well 
as more detailed reporting of product features implementation, would both 
bring some real benefits.  Some improvements *are* achievable, I think.  At 
the same time, Alex's comments on the pragmatic constraints that we would 
encounter in dealing with the real-world companies and the marketplace 
resonate with my own experiences.

This should go on the agenda of QA, to sort through the issues and resolve 
what improvements can be made (to spec contents, processes, or elsewhere), 
that will have some likelihood of success.

Some more comments follow (first about Karl's comments)...

>         I think also that a company or a developper claiming its support 
> to a W3C technology should do it but with a complete description of 
> what's implemented and what's not. it means having a list of all 
> supported elements and for each element the supported attributes.
>
>[...]
>How can we formulate such a thing?

This is the crux of a design issue, for the QA activity to resolve.  It 
needs to distinguish between normative content in a formal conformance 
clause, and helpful informative product-feature reporting -- both are useful.

>Do you want this as a developper, as a user, as an implementor?

Clearly it is the user (consumer of the technology) who benefits the 
most.  However, it would also useful to developers who might be trying to 
achieve interoperability with other implementations.

>How the list should be presented?

Another design issue for QA.

At 11:44 AM 10/8/2001 -0600, Alex Rousskov wrote:
>On Mon, 8 Oct 2001, Karl Dubost wrote:
>[...]
>In real world, conformance claims are mostly marketing tools.

I agree with this.  The claims are often made without any regard for 
specific meaning or accuracy.

>You can
>develop an elaborate set of rules how to claim conformance, but most
>implementors will ignore them, especially if
>         - the rules are not enforced

I am not suggesting that we *should* do this, but there has been some 
discussion of legal mechanisms, around trademark branding (of W3C specs), 
to control product claims.  I have seen similar things in other venues -- 
what sort of statements can/cannot be made by users of copyrighted (but 
freely available) test suites.  Reference to W3C branding and hints toward 
legal restraints is made in:

http://www.w3.org/Guide/Conformance.html

This (legally constraining users of copyrighted W3C specs) would be one 
extreme.  Doing nothing is the other extreme.

An interesting possible middle-ground model for enforcing rules happened in 
some recent WG work -- the members of a WG will often enforce the rules 
themselves.  For example, one company was violating W3Cs PR pre-approval 
policies, and the other companies brought the complaint.  The situation was 
resolved.

>         - the rules are difficult to obey/support

We should try to make it simple, but some cases are inherently 
difficult.  UAAG 1.0 (section 3) very carefully defines a fairly 
complicated set of rules about conditional conformance.  It looks daunting, 
but in fact it is deterministic and do-able.  (Some tools and automation -- 
e.g., conditional interactive forms -- could make it a little less daunting.)

>         - the claimed conformance status is difficult to verify

Test suites are needed, at the least.

>         - real world conditions mandate violation of the rules
>           (e.g., for compatibility with broken but popular agents)

This one is tough to deal with -- market forces rule.  (But see, e.g., the 
debate on WASP, referenced from the QA page.)


>Thus, I would not recommend wasting time on developing a complex
>system of conformance rules and procedures.

I don't think "complex" is necessary for most of the W3C standards, 
especially if the specs are well-written (e.g., avoiding unneeded 
optionality and discretion).  There will be some exceptions.

>Something like IETF
>conditional/unconditional conformance levels are good enough. When I
>see a "conforms to XYZ" claim, I take it with a grain of salt. It is
>still useful information for me (at least they know about and probably
>read the XYZ standard),

We have all heard "XYZ-compliant" claims that are totally vacuous.  The 
claimant is at the XYZ conference, and knows that he/she must say that, and 
has no idea what it even means(!).

>but I would not rely on that claim to be 100%
>true or accurate.

I agree with Karl, that the situation can be improved by at least some 
documentation of implemented features, performance on test suites, etc....

At 08:13 PM 10/8/2001 +0200, Karl Dubost wrote:
>>[...]
>Oh no, it's no a system to claim more conformance. Conformance is still an 
>open  issue to be solved. You can just see it as a way to declare what's 
>and what's not implemented inside a product.
>
>I think, people would be happy, to have such a list when they are using a 
>product. At that time, people claim: "I implement fooML", but it doesn't 
>mean anything for users, because you can implement only <tag_001> of the 
>fooML specification which contains almost 100 tags.
>
>If people implementing specs claim "I implement fooML and this is the list 
>of  implemented features as a list of elements and a list of attributes 
>for each individual element."
>
>This claim doesn't say anything about conformance. Is the feature right 
>implemented, is it compatible with other tools, etc? It's just a list of 
>what it's done and what's not. IMHO, it's an improvement, because now we 
>don't know at all what's inside a product.

It is necessary for us to distinguish between formal conformance claims 
(e.g., relative to section 3 or UAAG 1.0, or Appendix G of SVG), and 
informative material, but I think we need to address both.  The informative 
material can take many forms, by the way -- see for example,

http://www.w3.org/Graphics/SVG/Test/BE-ImpStatus.html

which was put together by SVG WG members, detailing leading products' 
results against 127 Basic Effectivity ("survey-level") tests which span the 
functionality of the standard (but do not probe it in fine detail).

Finally, back to some comments of Alex's:

At 12:55 PM 10/8/2001 -0600, Alex Rousskov wrote:
>[...]
>I can see only one way this system may be widely used in practice.
>That way is providing an extremely-easy-to-use form with checkboxes
>that developers can use to create a "we support these features"
>snapshot.

Yes.  If the utility of such information is agreed by all of us, then it 
points out some work for the QA activity and/or the individual WGs.  The 
barriers are, "who does the work", and dealing with market-force inertia, 
as Alex further points out:


>I think it is unrealistic to expect most developers to create and
>maintain a very detailed and up-to-date list of supported attributes
>for each individual element. [...]
>There is an incentive for the user. There is little incentive for the
>developer/companies [...]

Such pragmatic constraints define the task for those who want to pursue 
this issue:  can we find a middle ground (for both formal conformance and 
informative material) that is both achievable on the one hand, and useful 
on the other?  This ought to make for a lively topic in the QA groups.

-Lofton.

*******************
Lofton Henderson
1919 Fourteenth St., #604
Boulder, CO   80302

Phone:  303-449-8728
Email:  lofton@rockynet.com
*******************
Received on Tuesday, 9 October 2001 11:44:22 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:13:58 GMT