misuse of "conformance testing"?

A conformance test suite is one that's definitive, no?
i.e. if you pass all the tests, then you conform.

"Conformance testing ... is testing to *determine* whether a system
meets some specified standard."
http://en.wikipedia.org/wiki/Conformance_testing

Note my emphasis on *determine*.

The XML test suite is not like that; you can pass
all the tests and still not conform to the spec.
And yet it's labelled "Extensible Markup Language (XML) Conformance Test
Suites"
  -- http://www.w3.org/XML/Test/

I notice this because the RDFa test suite
 http://www.w3.org/2006/07/SWD/RDFa/testsuite/
says "The test suite provides for an RDFa conformance test"
and quotes a QA faq:

"... testing what is formally required in a specification in order to
verify whether an implementation conforms to its specifications.
Conformance testing does not focus on performance, usability, the
capability of an implementation to stand up under stress, or
interoperability; nor does it focus on any implementation-specific
details not formally required by the specification"
http://www.w3.org/QA/WG/2005/01/test-faq#why

That FAQ isn't signed, by the way.
http://www.w3.org/Provider/Style/SignIt

Feedback is directed to www-qa-wg@w3.org, but
http://www.w3.org/QA/WG/ says "The QA Working Group is now Closed".

In contrast to the RDFa and XML test suites, the OWL
test cases document is careful to say that it is *not*
a conformance test suite:

"However, the test cases do not constitute a conformance test suite for
OWL, since they are silent on several important issues. This document
cannot be considered a complete specification of OWL."
 -- http://www.w3.org/TR/owl-test/#scope


-- 
Dan Connolly, W3C http://www.w3.org/People/Connolly/
D3C2 887B 0F92 6005 C541  0875 0F91 96DE 6E52 C29E

Received on Friday, 2 March 2007 18:01:13 UTC