RE: [www-qa] Re: Conformance and Implementations

I guess I don't understand why 2 tests need to be developed to get the 
issue resolved.  Why not just ask the question as to what was intended and 
issue an errata.  The tests should interpret what is in the spec now.  I 
would argue that if a spec does not prohibit an unforeseen way to 
accomplish something, it must be allowed.  Thus, the permissive test should 
be developed and only changed if an errata re-specified this requirement in 
more stringent terms.

At 12:56 PM 10/19/01 -0600, Arnold, Curt wrote:
> > I strongly disagree with this philosophy.  Testing is
> > difficult enough
> > without making the tester test for someone else's perceived
> > interpretation.  In addition, this would not foster
> > interoperability.   The
> > sooner the tests reflect the accurate requirements of the
> > standard, the
> > sooner we have conforming implementations and interoperable products.
>
>Let me give you a specific scenario currently happening in the DOM test 
>suite that explains what I was trying to get at and hopefully that will 
>clarify what I was trying to express.
>
>
>In DOM Level 2, the DocumentType interface 
>(http://www.w3.org/TR/DOM-Level-2-Core/core.html#ID-412266927) has a 
>systemId property, the full description of the property is:
>
>The system identifier of the external subset.
>
>If the XML file has the following document type definition:
>
><!DOCTYPE staff SYSTEM 'staff.dtd'>
>
>Most DOM implementations return 'staff.dtd', however one implementation 
>converts the relative URL to a absolute URL and returns something like 
>'file:///c:/mydrive/mydir/staff.dtd'.
>
>Resolving the relative URL to an absolute URL was not required by the 
>description but was not explicitly prohibited either.
>
>A submitted test compensated for the absolutization by only comparing the 
>section after the last slash.  A test that all the implementation 
>pass.  Since I see that as reflecting permissive reading of
>the expected behavior, I could submit a comparable test that does a strict 
>string comparison, that all but one of the implementations pass and raise 
>the issue to the WG for resolution.
>
>The actions that could be expected by the Work Group are:
>
>A) Sustain the stricter test and issue an errata to the recommendation 
>indicating that an implicit absolutization of the systemID was not an 
>intended behavior.  The implementor who did do the
>absolution would be expected to modify their implementation to bring in to 
>conformance with the stricter test which has now been established as 
>expressing the intent and most common interpretation of
>the recommendation.
>
>B) Veto the stricter test and either issue an errata clarifying the 
>allowable behavior or dismiss the stricter test as an unreasonable 
>interpretation of the recommendation.
>
>In this particular case, I think the best resolution is A, however B is 
>better than leaving the issue unresolved.  Without writing explicit tests 
>and getting an explicit decision from the WG, the
>issue doesn't get resolved.

****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Friday, 19 October 2001 16:38:39 UTC