Re: misuse of "conformance testing"?

At 5:39 PM -0600 3 03 2007, Dan Connolly wrote:
>On Sat, 2007-03-03 at 00:58 -0500, david_marston@us.ibm.com wrote:
>>  >A conformance test suite is one that's definitive, no?
>>  >i.e. if you pass all the tests, then you conform.
>>
>>  Sorry, Dan. The QAWG was resolutely unanimous in the attitude that it
>>  is impractical to produce a test suite that proves conformance. Some
>>  drafts of the QA documents included a requirement that the WGs issue
>>  a statement to the effect that passing all tests in a test suite
>>  does not prove conformance.
>
>Ah. good.
>
>>  However, a conformance test suite can prove that an implementation
>>  does *not* conform, which can be useful.
>
>Hang on; are you using "conformance test suite" in the conventional
>way?

Whose convention?

<quote
cite="http://www.itl.nist.gov/div897/ctg/software.htm">

Conformance Test Suite Software

Conformance tests capture the technical description of a
specification and measure whether a product faithfully implements the
specification. The testing provides developers, users, and
purchasers, with increased levels of confidence in product quality
and increases the probability of successful interoperability.
</quote>

The result is heightened confidence.  A shift in probabilities.

I didn't emphasize 'measure,' because I think the text stands on its
own. I find nothing in the above statement that implies the question
of conformance is reduced to binary by the application of the suite.

The initial assumption

<quote
cite="http://lists.w3.org/Archives/Public/www-qa/2007Mar/0004.html">

A conformance test suite is one that's definitive, no?
i.e. if you pass all the tests, then you conform.

</quote>

.. is rash and unsupportable in the real world.

In particular, you note

<quote
cite=ibid>

"Conformance testing ... is testing to *determine* whether a system
meets some specified standard."
http://en.wikipedia.org/wiki/Conformance_testing

Note my emphasis on *determine*.

</quote>

Yes, the emphasis is yours and it is an over-emphasis. You are
reading precision into this use of this term that isn't there in the
intent of the author. The use of 'to determine' in this case does not
mean that the question at hand or the learning accomplished is binary.

Any group of tests where the primary function of the tests in the
collection is to shed light on the conformance of an instance to a
type-spec can properly be termed a "conformance test suite."

It doesn't have to be exhaustive or exact.

Al

>
>>   Buyers want to insist that
>>  the product they are buying must conform to the specs, but they
>>  should settle for insisting that the product passes all the tests in
>>  the conformance test suite (subject to permissible variability, and
>>  a good test suite can be filtered along those lines). The facts
>>  uncovered by a conformance test suite, incomplete though it may be,
>
>That's a contradiction in terms. If it's a conformance test
>suite, it can't be incomplete.
>
>>  aid in purchasing decisions and other real-world assessment
>>  activities.
>>
>>  As you discovered in the Test FAQ [1, probably the words of Patrick
>>  Curran], the term "conformance testing" is defined, and the term
>>  "conformance test suite" is implied. Despite the futility of
>>  developing a complete conformance test suite, it is useful to label
>>  incomplete suites as conformance test suites so that we understand
>>  their purpose.
>
>Is that a typo? Or are you really saying that it's useful
>to call them conformance test suites, even though they
>don't fit the conventional definition of the term?
>
>>   Therefore, suites such as the XML Conformance Test
>>  Suite are properly labeled.
>
>No, they're not.
>
>If W3C uses the term "conformance test suite", buyers
>will quite reasonably read the conventional meaning,
>which the XML test suite doesn't fulfill. It's not useful
>for W3C to go against the conventional usage this way.
>
>Try something... edit the wikipedia article
>   http://en.wikipedia.org/wiki/Conformance_testing
>so that it says that the XML test suite is a good
>example of a conformance test suite, even though
>it's incomplete. See how long the edit lasts.
>
>
>>  .................David Marston
>>
>>  [1]http://www.w3.org/QA/WG/2005/01/test-faq
>--
>Dan Connolly, W3C http://www.w3.org/People/Connolly/
>D3C2 887B 0F92 6005 C541  0875 0F91 96DE 6E52 C29E

Received on Sunday, 4 March 2007 00:55:08 UTC