RE: misuse of "conformance testing"?

Dan,
 
Thank you for this hint. I have to admit that I am 
not sure 100% by myself; especially regarding QA issues, 
I'm always open to learn, though.
 
Now, IMHO our intention was to create something like
a conformance test, viz. defining test assertions on
which one can test if her implementation _does_ conform
to RDFa. This is the focus of the Test Suite rather than
i14y, performance, etc.
 
In case you've got some better terminology, let us know ;)
 
Cheers,
       Michael
 
----------------------------------------------------------
 Michael Hausenblas, MSc.
 Institute of Information Systems & Information Management
 JOANNEUM RESEARCH Forschungsgesellschaft mbH
 Steyrergasse 17, A-8010 Graz, AUSTRIA

 <office>
    phone: +43-316-876-1193 (fax:-1191)  
   e-mail: michael.hausenblas@joanneum.at
      web: http://www.joanneum.at/iis/ <https://webmail.joanneum.at/exchweb/bin/redir.asp?URL=http://www.joanneum.at/iis/> 

 <private>
   mobile: +43-660-7621761
      web: http://www.sw-app.org/ <https://webmail.joanneum.at/exchweb/bin/redir.asp?URL=http://www.sw-app.org/> 
----------------------------------------------------------


________________________________

From: Dan Connolly [mailto:connolly@w3.org]
Sent: Fri 2007-03-02 19:01
To: www-qa@w3.org; www-qa-wg@w3.org
Cc: Hausenblas, Michael
Subject: misuse of "conformance testing"?



A conformance test suite is one that's definitive, no?
i.e. if you pass all the tests, then you conform.

"Conformance testing ... is testing to *determine* whether a system
meets some specified standard."
http://en.wikipedia.org/wiki/Conformance_testing

Note my emphasis on *determine*.

The XML test suite is not like that; you can pass
all the tests and still not conform to the spec.
And yet it's labelled "Extensible Markup Language (XML) Conformance Test
Suites"
  -- http://www.w3.org/XML/Test/

I notice this because the RDFa test suite
 http://www.w3.org/2006/07/SWD/RDFa/testsuite/
says "The test suite provides for an RDFa conformance test"
and quotes a QA faq:

"... testing what is formally required in a specification in order to
verify whether an implementation conforms to its specifications.
Conformance testing does not focus on performance, usability, the
capability of an implementation to stand up under stress, or
interoperability; nor does it focus on any implementation-specific
details not formally required by the specification"
http://www.w3.org/QA/WG/2005/01/test-faq#why

That FAQ isn't signed, by the way.
http://www.w3.org/Provider/Style/SignIt

Feedback is directed to www-qa-wg@w3.org, but
http://www.w3.org/QA/WG/ says "The QA Working Group is now Closed".

In contrast to the RDFa and XML test suites, the OWL
test cases document is careful to say that it is *not*
a conformance test suite:

"However, the test cases do not constitute a conformance test suite for
OWL, since they are silent on several important issues. This document
cannot be considered a complete specification of OWL."
 -- http://www.w3.org/TR/owl-test/#scope


--
Dan Connolly, W3C http://www.w3.org/People/Connolly/
D3C2 887B 0F92 6005 C541  0875 0F91 96DE 6E52 C29E

Received on Friday, 2 March 2007 21:33:43 UTC