Re: misuse of "conformance testing"?

On Sat, 2007-03-03 at 00:58 -0500, david_marston@us.ibm.com wrote:
> >A conformance test suite is one that's definitive, no?
> >i.e. if you pass all the tests, then you conform.
> 
> Sorry, Dan. The QAWG was resolutely unanimous in the attitude that it
> is impractical to produce a test suite that proves conformance. Some
> drafts of the QA documents included a requirement that the WGs issue
> a statement to the effect that passing all tests in a test suite
> does not prove conformance.

Ah. good.

> However, a conformance test suite can prove that an implementation
> does *not* conform, which can be useful.

Hang on; are you using "conformance test suite" in the conventional
way?

>  Buyers want to insist that
> the product they are buying must conform to the specs, but they
> should settle for insisting that the product passes all the tests in
> the conformance test suite (subject to permissible variability, and
> a good test suite can be filtered along those lines). The facts
> uncovered by a conformance test suite, incomplete though it may be,

That's a contradiction in terms. If it's a conformance test
suite, it can't be incomplete.

> aid in purchasing decisions and other real-world assessment
> activities.
> 
> As you discovered in the Test FAQ [1, probably the words of Patrick
> Curran], the term "conformance testing" is defined, and the term
> "conformance test suite" is implied. Despite the futility of
> developing a complete conformance test suite, it is useful to label
> incomplete suites as conformance test suites so that we understand
> their purpose.

Is that a typo? Or are you really saying that it's useful
to call them conformance test suites, even though they
don't fit the conventional definition of the term?

>  Therefore, suites such as the XML Conformance Test
> Suite are properly labeled.

No, they're not.

If W3C uses the term "conformance test suite", buyers
will quite reasonably read the conventional meaning,
which the XML test suite doesn't fulfill. It's not useful
for W3C to go against the conventional usage this way.

Try something... edit the wikipedia article
  http://en.wikipedia.org/wiki/Conformance_testing
so that it says that the XML test suite is a good
example of a conformance test suite, even though
it's incomplete. See how long the edit lasts.


> .................David Marston
> 
> [1]http://www.w3.org/QA/WG/2005/01/test-faq
-- 
Dan Connolly, W3C http://www.w3.org/People/Connolly/
D3C2 887B 0F92 6005 C541  0875 0F91 96DE 6E52 C29E

Received on Saturday, 3 March 2007 23:40:32 UTC