RE: [www-qa] Re: Conformance and Implementations

This is a very tricky point.  In theory, the developers of the test suite 
interpret the standard.  At NIST, we've developed many test suites.  I've 
always said that since most specifications are ambiguous, at best, and 
contradictory at worst, that the test suite then becomes the (official 
interpretation of the) standard.  We've come across standards developers 
who insist that they meant one thing, but if the standard is clear, we test 
for what the standard says, not what the specifiers meant.

So the only confusion comes when something is ambiguous.  In that case, I 
would agree that the standards body (in this case W3C) should make the 
interpretation.  However, the tester should not test for something that is 
not clear in the standard.  The standard needs to be revised to reflect 
the  intent.  Only after this occurs, can a test suite test for that 
requirement.  Until then, the test for that requirement should be 
withdrawn.  Standards developers must stand behind the wording in the 
standard.

I do believe that interpretations need to be "the best".  It's not fair to 
implementers who, many times, have spent long hours interpreting and 
implementing the spec to change the rules on them in the middle of the game.

Mark




At 06:43 PM 10/18/01 -0700, Kirill Gavrylyuk wrote:
>David,
>but would you agree that, while third party may do much better job in 
>creating tests, final decision on spec interpretation should belong to W3C 
>WG, regardless of whether it is the "best interpretation" or not.
>
>Interpretation can not be "the best", it should just come from single 
>source to avoid chaos.
>
>
>-----Original Message-----
>From: David_Marston@lotus.com [mailto:David_Marston@lotus.com]
>Sent: Thursday, October 18, 2001 6:05 PM
>To: www-qa@w3.org
>Subject: [www-qa] Re: Conformance and Implementations
>
>
>
>Dimitris wrote:
> >3. It is not clear who (normatively speaking) does the best job in
> >interpreting the specifcation in question ((which is why the DOM TS ML
> >Schema is generated directly from the DOM specs). Is it the WG who
> >wrote the spec? Is it a trusted third party? Is it the member companies?
> >I believe this to be the most serious problem.
>
>I agree completely. Specifically, a third party can do a better job than
>the WG by trying to deduce test assertions from the written normative
>documents (at CR stage or later). Inevitably, the WG reaches a consensus
>or "understanding" on some fine points that the Recommendation does not
>convey. An attempt to write test cases can expose such gaps just as an
>attempt to develop a working implementation would do.
>.................David Marston

****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Friday, 19 October 2001 10:01:58 UTC