Re: Jena implementation report plans

On September 8, Jim Hendler writes:
> 
> At 11:33 AM +0100 9/7/03, Ian Horrocks wrote:
> >On September 5, Jim Hendler writes:
> >>
> >>  At 3:29 PM +0100 9/4/03, Dave Reynolds wrote:
> >>
> >>  [snip]
> >>
> >>  >Approximately 10 test cases could be usefully reformulated this way.
> >>  >
> >>  >Possible responses to this comment include:
> >>  >1. Modify some of test cases to this simple-conclusion style.
> >>  >2. Augment the test cases by duplicates in this style.
> >>  >3. Ignore it and leave the test cases as is.
> >>  >
> >>  >Dave
> >>
> >>  My preference would be 2 or 1 in that order - anything that makes it
> >>  easier for people to test implementations and to help them understnad
> >>  how to build tools seems like a good idea to me!
> >
> >I am opposed to 2 and strongly opposed to 1.
> >
> >My assumption is that the tests are intended to illustrate features of
> >the language and/or potential difficulties in implementation. Speaking
> >as an implementor, these kinds of "tricky" test are just what we need
> >- in fact we need more/harder tests of this kind. I don't see how
> >modifying tests to make them easier to pass is of any real help to
> >implementors - unless our intention is to fool people into believing
> >that it is easier to implement an OWL reasoner than is really the
> >case.
> >
> >Even adding duplicate easier versions of tests seems to be
> >misguided. If we want to "encourage" implementors, we could simply add
> >lots of trivial tests that everyone can easily pass (just to be clear,
> >I am not suggesting this!). And where does this end - do we add
> >multiple versions of every test, each carefully tuned so that it can
> >be passed by a given implementation?
> >
> >Ian
> >
> >
> 
> Ian, as usual you and I seem to see diametrically opposite solutions 
> to the same problem -- my idea is that if we have two tests (note, my 
> preferense was the one where we create EXTRA tests) so that people 
> can more clearly see what the tests are about -- we've had several 
> times where our code failed a test, and we wasted a long time trying 
> to debug the wrong thing - because many of our tests test two or 
> three things all at the same time -- by have a simple and complex 
> version of the same test, it can make it much easier for people 
> who've never implemented a reasoner before to understand where the 
> complexities lie -- it would be easy to generate a large number of 
> worthless tests, but it would also be easy to only have a few tests 
> that were so hard it wasn't clear if passing them demonstrated 
> generality - I prefer anything that makes our test set more 
> meaningful to developers -- the goal is to help them build OWL 
> software, not to "test" them in the sense that a final exam tests my 
> students (and my tests are known for being tough in the latter
> case!)

I don't have a problem with adding extra tests that are designed to
help implementors, but I am doubtful if adding tests that can be
passed by certain styles if incomplete implementation is really
achieving that objective - better for them to be made aware of the
areas in which their implementation is incomplete. Removing tests that
highlight tricky inferences (your second choice) certainly doesn't
help implementors!

These kinds of "final exam" test (small but hard examples) are
invaluable for implementors - I speak not only from my own experience
but also from conversations with other implementors. Such a test
typically highlights some area of the semantics where implementations
may easily get it wrong. In the case where this does happen, the tests
are small enough that it is easy for implementors to work through them
and see where their implementation went wrong.

Ian


>   -JH
> 
> -- 
> Professor James Hendler				  hendler@cs.umd.edu
> Director, Semantic Web and Agent Technologies	  301-405-2696
> Maryland Information and Network Dynamics Lab.	  301-405-6707 (Fax)
> Univ of Maryland, College Park, MD 20742	  *** 240-277-3388 (Cell)
> http://www.cs.umd.edu/users/hendler      *** NOTE CHANGED CELL NUMBER ***
>

Received on Monday, 15 September 2003 08:29:17 UTC