Re: Manual Rewriting and Passing Entailments

On Thu, 11 Sep 2003, Jeremy Carroll wrote:

>
> Summary:
> Do systems need a fully automated test harness to pass a test?
>
>
>
> I was chatting with Dave Reynolds about what is expected to pass an
> entailement test.
>
> The tests are expressed as
>
> Graph1 entails Graph2
>
> In practice many APIs (including ours) do not directly support such an
> operation.
>
> Hence Dave automatically transforms Graph2 into a query which he can then
> execute againsts Graph1, and pass the test.
>
> That looks fine to me.
>
> For some of the tests, he has a more complex query rewrite that he does
> manually, and then passes the test. I am discouraging him from reporting such
> tests as passed. (These reflect the lack of support for the comprehension
> axioms - the query rewrite essentially compensates for this).
>
> ===
>
> What are other people doing? How much manual and/or automatic rewrite do
> people do?

All the rewriting for my testing is done automatically. Each test is
translated to a single problem in TPTP which can then be checked for
consistency. There is a certain amount of manual work that needs to be
done in terms of running the prover over the resulting problem and
analysing the results, but this is just plumbing.

I have a translator which takes an owl ontology and translates to a
collection of FO axioms. e.g.

ont -> ax1, ax2, ax3,....axn

For (in)consistency tests, we then see if:

ax1 /\ ax2 /\ ... axn

is (in)consistent

For entailment, if the test involves

ontA entails ontB

then we translate to:

axA1 /\ axA2 /\ ... /\ axAn /\ ~(axB1 /\ axB2 /\ ... axBn)

e.g. conjoin the translatino of A with negation of B and look for an
inconsistency. This current approach ignores annotations (although this
could be done via some simple lookups). I am not tackling negative
entailment tests.

I would personally be wary of a system that required manual rewrites to
pass a test, and would not really consider these as passes.

	Sean

-- 
Sean Bechhofer
seanb@cs.man.ac.uk
http://www.cs.man.ac.uk/~seanb

Received on Thursday, 11 September 2003 05:10:05 UTC