Re: Conformance and Implementations

Lofton Henderson wrote:
>Until the spec is corrected with an erratum, the test suite should not 
>attempt to impose an interpretation.  Nothing short of a consensus
>erratum process addresses this problem definitively.

The idea of writing a contentious test is to force an appeal and a definitive ruling and/or errata when a particular issue is ambigious, not to usurp the authority of the WG but to ensure that the issue is addressed by the WG.  Before the acceptance of a test suite, the WG should review and confirm any test that one or more implementations fail or the test developers identify as debatable.

Since the development of a test suite may raise many of these minor issues, they are probably most effectively as a block of issues during the test suite ratification process.  The contentious tests serve roughly the same role as the last call list, issues that need to be addressed before the test suite is accepted.

Opinions from individual WG members are not sufficient.  A good example of how this could go wrong is a thread (unfortunately spanning the Xerces-J and www-dom mailing lists) on a test on removing an event listener during an event dispatch.

The following URL's are starting points on the www-dom mailing lists on the topic
http://lists.w3.org/Archives/Public/www-dom/2001JulSep/0225.html and http://lists.w3.org/Archives/Public/www-dom/2001JulSep/0246.html.

In a message from Arnald Le Hors (on the Xerces-J mailing list near 11 Aug 2001)
> You're right there, that's what the spec currently says. But the spec is
> wrong, though. It is meant to say the opposite. I just double-checked
> with Joe Kesselman who was very much involved in the development of that
> spec and who implemented it in Xerces and he confirmed my fears.
> I will bring it up to the DOM Working Group, I would expect an erratum
> to be published.

Actually, the Xerces-J implementation reflected the formulation in a working draft that was superceded and the working group had explicitly and intentionally changed the behavior a working draft or two before recommendation.  In this case, the test reflected the intention of the WG where an existing implementation and two very knowledgable practitioners recollections were faulty.

If the test group is supposed to avoid writing contentious tests, the test would have been rejected even though it, not an existing implementation and knowledgable WG members, reflected both the text of the specification and the intent of the WG.