Re: Jena implementation report plans

At 11:33 AM +0100 9/7/03, Ian Horrocks wrote:
>On September 5, Jim Hendler writes:
>>
>>  At 3:29 PM +0100 9/4/03, Dave Reynolds wrote:
>>
>>  [snip]
>>
>>  >Approximately 10 test cases could be usefully reformulated this way.
>>  >
>>  >Possible responses to this comment include:
>>  >1. Modify some of test cases to this simple-conclusion style.
>>  >2. Augment the test cases by duplicates in this style.
>>  >3. Ignore it and leave the test cases as is.
>>  >
>>  >Dave
>>
>>  My preference would be 2 or 1 in that order - anything that makes it
>>  easier for people to test implementations and to help them understnad
>>  how to build tools seems like a good idea to me!
>
>I am opposed to 2 and strongly opposed to 1.
>
>My assumption is that the tests are intended to illustrate features of
>the language and/or potential difficulties in implementation. Speaking
>as an implementor, these kinds of "tricky" test are just what we need
>- in fact we need more/harder tests of this kind. I don't see how
>modifying tests to make them easier to pass is of any real help to
>implementors - unless our intention is to fool people into believing
>that it is easier to implement an OWL reasoner than is really the
>case.
>
>Even adding duplicate easier versions of tests seems to be
>misguided. If we want to "encourage" implementors, we could simply add
>lots of trivial tests that everyone can easily pass (just to be clear,
>I am not suggesting this!). And where does this end - do we add
>multiple versions of every test, each carefully tuned so that it can
>be passed by a given implementation?
>
>Ian
>
>

Ian, as usual you and I seem to see diametrically opposite solutions 
to the same problem -- my idea is that if we have two tests (note, my 
preferense was the one where we create EXTRA tests) so that people 
can more clearly see what the tests are about -- we've had several 
times where our code failed a test, and we wasted a long time trying 
to debug the wrong thing - because many of our tests test two or 
three things all at the same time -- by have a simple and complex 
version of the same test, it can make it much easier for people 
who've never implemented a reasoner before to understand where the 
complexities lie -- it would be easy to generate a large number of 
worthless tests, but it would also be easy to only have a few tests 
that were so hard it wasn't clear if passing them demonstrated 
generality - I prefer anything that makes our test set more 
meaningful to developers -- the goal is to help them build OWL 
software, not to "test" them in the sense that a final exam tests my 
students (and my tests are known for being tough in the latter case!)
  -JH

-- 
Professor James Hendler				  hendler@cs.umd.edu
Director, Semantic Web and Agent Technologies	  301-405-2696
Maryland Information and Network Dynamics Lab.	  301-405-6707 (Fax)
Univ of Maryland, College Park, MD 20742	  *** 240-277-3388 (Cell)
http://www.cs.umd.edu/users/hendler      *** NOTE CHANGED CELL NUMBER ***

Received on Monday, 8 September 2003 18:58:44 UTC