Re: misuse of "conformance testing"?

>A conformance test suite is one that's definitive, no?
>i.e. if you pass all the tests, then you conform.

Sorry, Dan. The QAWG was resolutely unanimous in the attitude that it
is impractical to produce a test suite that proves conformance. Some
drafts of the QA documents included a requirement that the WGs issue
a statement to the effect that passing all tests in a test suite
does not prove conformance.

However, a conformance test suite can prove that an implementation
does *not* conform, which can be useful. Buyers want to insist that
the product they are buying must conform to the specs, but they
should settle for insisting that the product passes all the tests in
the conformance test suite (subject to permissible variability, and
a good test suite can be filtered along those lines). The facts
uncovered by a conformance test suite, incomplete though it may be,
aid in purchasing decisions and other real-world assessment
activities.

As you discovered in the Test FAQ [1, probably the words of Patrick
Curran], the term "conformance testing" is defined, and the term
"conformance test suite" is implied. Despite the futility of
developing a complete conformance test suite, it is useful to label
incomplete suites as conformance test suites so that we understand
their purpose. Therefore, suites such as the XML Conformance Test
Suite are properly labeled.
.................David Marston

[1]http://www.w3.org/QA/WG/2005/01/test-faq

Received on Saturday, 3 March 2007 05:58:32 UTC