Re: [www-qa] Re: Conformance and Implementations

Here's a thought: Requiremnet of conformance to what?

Standard -> Assertions -> Tests
   |            |           |
   |            |           |
   ?------------?-----------?
                |
                ?
    Conformance Requirements -> Service/product


Which is normative?
The Standard must be, always...
but...
conformance to what?

One method is to take a 'Venn diagram' approach to CR - define a product or
service of some sort that will actually be useful to people (rather than a more
abstract standard). Work out the conformance requirements- which may overlap
with several normative standards.

The Assertions define a set of behaviours that the service must exhibit to be
conformant. The tests act as an _indicator_ of compliance. That is - a test
suite, as discussed before, can not guarantee 100% conformance to a standard.
There are usually mandatory but untestable features.

So conformance requirements could be tied to something other than the base
standard.

Conformance testing - is it arbitrary? is it just a matter of where we draw the
line? or does it have inviolable limits?


Andrew


 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 Dr. Andrew Thackrah        The Open Group
 Software Development       Apex Plaza, Forbury Rd    T H E
 Group                      Reading, RG1 1AX          O P E N
 a.thackrah@opengroup.org   United Kingdom            G R O U P
 http://www.opengroup.org   +44 (0)118 950 8311
 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-




On Oct 19, 10:03am, Mark Skall wrote:
> Subject: RE: [www-qa] Re: Conformance and Implementations
> This is a very tricky point.  In theory, the developers of the test suite
> interpret the standard.  At NIST, we've developed many test suites.  I've
> always said that since most specifications are ambiguous, at best, and
> contradictory at worst, that the test suite then becomes the (official
> interpretation of the) standard.  We've come across standards developers
> who insist that they meant one thing, but if the standard is clear, we test
> for what the standard says, not what the specifiers meant.
>
> So the only confusion comes when something is ambiguous.  In that case, I
> would agree that the standards body (in this case W3C) should make the
> interpretation.  However, the tester should not test for something that is
> not clear in the standard.  The standard needs to be revised to reflect
> the  intent.  Only after this occurs, can a test suite test for that
> requirement.  Until then, the test for that requirement should be
> withdrawn.  Standards developers must stand behind the wording in the
> standard.
>
> I do believe that interpretations need to be "the best".  It's not fair to
> implementers who, many times, have spent long hours interpreting and
> implementing the spec to change the rules on them in the middle of the game.
>
> Mark
>
>
>
>
> At 06:43 PM 10/18/01 -0700, Kirill Gavrylyuk wrote:
> >David,
> >but would you agree that, while third party may do much better job in
> >creating tests, final decision on spec interpretation should belong to W3C
> >WG, regardless of whether it is the "best interpretation" or not.
> >
> >Interpretation can not be "the best", it should just come from single
> >source to avoid chaos.
> >
> >
> >-----Original Message-----
> >From: David_Marston@lotus.com [mailto:David_Marston@lotus.com]
> >Sent: Thursday, October 18, 2001 6:05 PM
> >To: www-qa@w3.org
> >Subject: [www-qa] Re: Conformance and Implementations
> >
> >
> >
> >Dimitris wrote:
> > >3. It is not clear who (normatively speaking) does the best job in
> > >interpreting the specifcation in question ((which is why the DOM TS ML
> > >Schema is generated directly from the DOM specs). Is it the WG who
> > >wrote the spec? Is it a trusted third party? Is it the member companies?
> > >I believe this to be the most serious problem.
> >
> >I agree completely. Specifically, a third party can do a better job than
> >the WG by trying to deduce test assertions from the written normative
> >documents (at CR stage or later). Inevitably, the WG reaches a consensus
> >or "understanding" on some fine points that the Recommendation does not
> >convey. An attempt to write test cases can expose such gaps just as an
> >attempt to develop a working implementation would do.
> >.................David Marston

>-- End of excerpt from Mark Skall

Received on Friday, 19 October 2001 11:19:20 UTC