Sesame compliance with RDF Test Cases of August 1

The following is an overview of Sesame's[3] compliance with the
test cases as mentioned in [0], along with editorial comments on
the format and presentation of the test cases.

Parser Test cases:
------------------

We have run Sesame's RDF parser from Sesame 0.9.2 against the new
RDF test cases of 25 July 2003[1]. Of the 170 APPROVED parser test
cases, 167 pass and 3 fail.

The 3 failing test cases are:

* rdfms-seq-representation/test001.rdf

   This test fails because Sesame's RDF parser generates
   (_:nodeId, <rdf:type>, <rdf:List>) triples for every bNode that
   is a list. This is conform the specification of 23 January 2003,
   but this has been changed in the editor's draft. The parser will
   be adapted to the new specs once they are officially released.

* rdf-charmod-literals-error001.rdf and
   rdf-charmod-literals-error002.rdf

   This is a known issue in Sesame's RDF parser: it accepts all
   unicode encodings and not just Normal Form C. From Peter F.
   Patel-Schneider's e-mail to rdf-comments [2] we conclude that
   some discussions are going about dropping this constraint from
   RDF. We're waiting for the outcome of this discussion before
   we're starting fixing this issue.

Entailment test cases:
----------------------

We have not yet run a full test run with the Entailment test cases
mentioned in [0]: to be honest the RDF Core WG caught us
off-guard, we were unaware that such a test case set was being
prepared until last friday. Negligence on our part I presume.

Having examined the test cases, we are somewhat surprised that in
fact most test cases seem to deal with borderline details of
datatype entailment, and that there are virtually no tests
whatsoever concerning more mainstream entailment issues, like
class- and property subsumption, cycles in hierarchies, domain and
range restrictions, etc.

Furthermore there are several test that require further
clarification: based on the premise and conclusion data sets it is
often confusing to understand why a particular entailment is or is
not valid.

As a point in case: the tests datatypes/test004(a/b) are both
mentioned as positive and negative tests. It is not explained why
this is so in [0].

The results presented here are not automatically tested, we have
examined the test cases manually, tried out those where we were
unsure how Sesame would react, and for the others the results are
our opinion on how Sesame _would_ react.

Of the 17 approved positive tests, Sesame (0.9.2) passes 7, fails
8, and 2 are unknown.

The 8 failed tests are:

  - datatypes/test007a -> test007b
  - datatypes/test004b -> test004c
  - datatypes/test005a -> test005b
  - datatypes/test003a -> test003b
  - datatypes/test003b -> test003a
  - datatypes/test008a -> test008b

    These all deal with related issues of comparing datatype values
    to each other. Since a lot of the work concerning datatype
    entailments seems to be more or less in flux, and a number of
    editorial changes are anticipated, we will not fix these  until
    a new release of the RDF Semantics WD.

- rdfms-seq-representation/empy -> test002
- rdfms-seq-representation/empy -> test004

    These cases demands that an empty graph
    entails that rdf:_1 is a containermembershipproperty, and a
    subproperty of rdfs:member. The Sesame inferencer is currently
    implemented in such a way that  the rdf:_1 (etc.) properties
    are only added to the inference process when a container is
    added to the repository. Thus, the inference is only made
    later.

- xmlsch-02/test001 -> test003

   According to this test a well-formed XML-Schema-datatyped
   literal entails an empty Literal node. We are unclear how this
   follows from the RDF MT.

The 2 unknown tests are those tests that do not present a
"conclusion" document, but merely state FALSE. We are unsure how
to interpret these tests.

Of the 17 approved negative tests, Sesame (0.9.2) passes 11, fails
1, and 3 unknown.

The 3 unknown tests are:

  - datatypes/test004a -> test004b
  - datatypes/test004b -> test004c

    Both these tests are also mentioned as positive tests,
    therefore it is not quite clear what the expected behaviour of
    the system in fact should be.

  - datatypes/test002 -> test002b
    It is simply unclear to us at this moment what aspect of
    entailment this test is supposed to capture.

Remark: for some reason two tests (statement-entailment/test001a
and statement-entailment/test002a) are listed twice. Sesame passes
the tests, but we are unsure how to interpret the double listing.

The 1 failed test is:

  - xmlsch-02/test002 -> test003

    According to the description the supplied literal is not
    well-formed, but it is not explained why. Clarification in the
    documentation of the test case might be useful.

Editorial remarks:

We feel that the current set of entailment test cases 
over-emphasizes the importance of datatype reasoning, and that in
many projects and practical applications, the importance of
correct subsumption reasoning is at least as high, if not much
higher. The set of test cases should reflect that.

Second, we feel that the application of the entailment test cases
is severely under-documented.

We trust this overview is of some help, and welcome your feedback
on our first (admittedly somewhat hurried) findings.

Regards,

Jeen Broekstra and Arjohn Kampman

References:
[0]http://www.w3.org/2001/sw/RDFCore/TR/WD-rdf-testcases-20030117/
[1]http://www.w3.org/2000/10/rdf-tests/rdfcore/
[2]http://lists.w3.org/Archives/Public/www-rdf-comments/2003JulSep/0152.html
[3]http://sesame.aidministrator.nl/

-- 
jeen.broekstra@aidministrator.nl
aidministrator nederland b.v.  -  http://www.aidministrator.nl/
prinses julianaplein 14-b, 3817 cs amersfoort, the netherlands
tel. +31-(0)33-4659987   fax. +31-(0)33-4659987

Received on Monday, 4 August 2003 11:05:54 UTC