RE: misuse of "conformance testing"?

Dan,
 
> Look at it this way: if I just build a piece of
> software that hard-codes all the answers to the
> tests, it will pass all the tests, right? But
> it won't actually do anything useful for inputs
> other than the ones in the test suite.

Thanks for this argument, though  IMHO one of us two 
starts with an incorrect precondition. Let me show you
what I mean:
 
The rational behind the Test Suite (TS) - that is btw the name, 
already ;) - is to show how a conforming RDFa implementation should
behave, viz. the TS is a set of  necessary, but not necessary and
sufficient conditions. Put in more formal words, partially 
using the terminology from [1]:
 
  conformantToRDFaSpec(?x) <- RDFaAwareAgent(?x) 
 
and _not_
 
  conformantToRDFaSpec(?x) <-> RDFaAwareAgent(?x) 
 
 
And, even if I am wrong with the above, my feeling tells me that
there is some wrong with your argumentation. This is due to that 
in your explanation - as far as I understand it - conformance
testing (in finite time) wouldn't be possible at all ;)
 
Cheers,
       Michael
 
[1] http://www.w3.org/2006/07/SWD/RDFa/#sec3 <http://www.w3.org/2006/07/SWD/RDFa/#sec3> 
 
----------------------------------------------------------
 Michael Hausenblas, MSc.
 Institute of Information Systems & Information Management
 JOANNEUM RESEARCH Forschungsgesellschaft mbH
 Steyrergasse 17, A-8010 Graz, AUSTRIA
----------------------------------------------------------


________________________________

From: Dan Connolly [mailto:connolly@w3.org]
Sent: Fri 2007-03-02 22:52
To: Hausenblas, Michael
Cc: www-qa@w3.org; www-qa-wg@w3.org
Subject: RE: misuse of "conformance testing"?



On Fri, 2007-03-02 at 22:33 +0100, Hausenblas, Michael wrote:
> Dan,
> 
> Thank you for this hint. I have to admit that I am
> not sure 100% by myself; especially regarding QA issues,
> I'm always open to learn, though.
> 
> Now, IMHO our intention was to create something like
> a conformance test, viz. defining test assertions on
> which one can test if her implementation _does_ conform
> to RDFa.

That's a *very* high bar. I wonder if you really mean
to meet it. I'm sure you mean to make a test suite that
can help to find lots of ways that an implementation
does *not* conform. But do you really mean for the
test suite to serve as a *complete* specification
of the language, such that the *only* way to pass
all the tests is to build a conforming implementation?

Look at it this way: if I just build a piece of
software that hard-codes all the answers to the
tests, it will pass all the tests, right? But
it won't actually do anything useful for inputs
other than the ones in the test suite.



>  This is the focus of the Test Suite rather than
> i14y, performance, etc.
> 
> In case you've got some better terminology, let us know ;)

Just call it a test suite. And say that it's purposes
are
 (a) to clarify issues that come up during the specification
development and
 (b) to assist with test-driven development of implementations.


--
Dan Connolly, W3C http://www.w3.org/People/Connolly/
D3C2 887B 0F92 6005 C541  0875 0F91 96DE 6E52 C29E

Received on Saturday, 3 March 2007 12:39:24 UTC