Re: [www-qa] Re: Conformance and Implementations

Comments inlined

On Friday, October 19, 2001, at 04:43  AM, Kirill Gavrylyuk wrote:

> David,
> but would you agree that, while third party may do much better job in 
> creating tests, final decision on spec interpretation should belong to 
> W3C WG, regardless of whether it is the "best interpretation" or not.
>
> Interpretation can not be "the best", it should just come from single 
> source to avoid chaos.
>
[dd] This is a very practical way to solve the issues that may be raised 
in interpreting specifications. It is the path we've followed in the DOM 
TS, letting the DOM WG be the normative reference for such disagreement. 
However, as David points out, there's a point to be made in letting a 
third party evaluate the spec and provide, if not a final 
interpretation, at least the test assertions.

>
> -----Original Message-----
> From: David_Marston@lotus.com [mailto:David_Marston@lotus.com]
> Sent: Thursday, October 18, 2001 6:05 PM
> To: www-qa@w3.org
> Subject: [www-qa] Re: Conformance and Implementations
>
>
>
> Dimitris wrote:
>> 3. It is not clear who (normatively speaking) does the best job in
>> interpreting the specifcation in question ((which is why the DOM TS ML
>> Schema is generated directly from the DOM specs). Is it the WG who
>> wrote the spec? Is it a trusted third party? Is it the member 
>> companies?
>> I believe this to be the most serious problem.
>
> I agree completely. Specifically, a third party can do a better job than
> the WG by trying to deduce test assertions from the written normative
> documents (at CR stage or later). Inevitably, the WG reaches a consensus
> or "understanding" on some fine points that the Recommendation does not
> convey. An attempt to write test cases can expose such gaps just as an
> attempt to develop a working implementation would do.
> .................David Marston
>
[dd] As far as your first sentence goes, it couldn't be said better. In 
addition to the rationale you mention, letting a third party draw up 
test assertions (ideally given a set of test assertion guidelines) makes 
clearer the distinction between specifiers of a technology and users of 
that technology. We all know the paradox of  the people who specify a 
particular technology rarely are those who are best at using it.

However, I think that is the job this trusted third party should do, 
mostly for practical reasons. Letting them assume responsibility for the 
interpretation of an entire specification is in my opinion far-fetched 
and non-practical.

Further to this, Alex points out:

On Thu, 18 Oct 2001, Kirill Gavrylyuk wrote:

while third party may do much better job in creating tests, final
decision on spec interpretation should belong to W3C WG,
regardless of whether it is the "best interpretation" or not.

I disagree. After the specs are frozen, there should be no _normative_
interpretations from W3C or any other body. Anybody can express their
opinions on what the spec means or what the specs meant to say, but
those opinions should be just that -- opinions.

[dd] I can't see that this is the safest way to do things. It means, 
among other things, disconnecting the specification from those who are 
responsible for it, and I don't think that's a good idea.

If specs need interpretation, it usually means that the specs are
ambiguous and broken. Such specs should be fixed, if possible, by
releasing new versions of the specs, and not by posting normative
opinions on WG mailing lists or W3C Web sites.

[dd] This is a good point. Specifications in need of interpretation are 
clearly lacking something. However, as pointed out by myself and others 
in the discussion that preceded the DOM TS Process document (please tell 
me if you need pointers, I'm offline and not on a very good connection), 
the test process can have a series of aims:

1. To test the implementations of the specification. This is the first 
and basic aim.
2. In cases where there is ambiguity, the specification itself is put to 
the test via the test suite. It is not a basic aim, but is part of the 
process. As this is bound to happen (for technical, conceptual or 
political issues, and hence needs to be resolved) it is safe to assume 
that there is need for a final interpretation, be it WG, community or 
otherwise.
3. Lack of consensus over the results of the test suite could be brought 
into the process of writing the specification itself; in our model the 
tests should ideally be written and run parallelly to writing the 
specification to identiify bottle-necks and problems as soon as possible 
as well as streamline the specification process itself.

Interpretation can not be "the best", it should just come from
single source to avoid chaos.

Ideally, there should be no need for interpretations.

[dd] Ideally, yes, in practice though, no, unfortunately.

In practice, many consider a self-organized chaos (e.g., nature,
democracy, or an open-source movement) to be a much more robust
architecture than a single source of failure (e.g., a dictatorship). As
it stands today, W3C is closer to a self-organized chaos.  I hope it
does not try to "simplify the problem" by becoming a dictatorship.

[dd] I don't think that's at stake. It's just a question of trying to 
assign ownership of specifications, which I think it is fair that it be 
assigned to those who wrote it. Except if all specifications can have 
some kind of super group as a trusted third party, which in turn acts as 
the normative instance. In any case, I can''t see how we could come away 
with having no ownership and hence interpretation at all. Speaking from 
my personal experience with the DOM TS, and I think this is adaptable to 
other test suites and specifications, I wouldn't have made it on my own 
without the DOM WG acting as final authority, in particular cases that 
needed to be resolved.

Best,

/Dimitris

$0.02,

Alex.

P.S. No normative interpretations are needed if there is no formal
      certification process endorsed by W3C, which is the model I root
      for.