[Fwd: Sesame compliance with RDF Test Cases of August 1]

Jan,

Just checking you saw this on rdf-comments.

There are detailed aspects of test cases that need your attention.

Brian

-------- Original Message --------
Subject: Sesame compliance with RDF Test Cases of August 1
Resent-Date: Mon, 4 Aug 2003 11:05:56 -0400 (EDT)
Resent-From: www-rdf-comments@w3.org
Date: Mon, 04 Aug 2003 17:05:50 +0200
From: Jeen Broekstra <jeen.broekstra@aidministrator.nl>
Organization: Aidministrator Nederland b.v.
To: www-rdf-comments@w3.org
CC: Dave Beckett <dave.beckett@bristol.ac.uk>,        Arjohn Kampman 
<arjohn.kampman@aidministrator.nl>



The following is an overview of Sesame's[3] compliance with the
test cases as mentioned in [0], along with editorial comments on
the format and presentation of the test cases.

Parser Test cases:
------------------

We have run Sesame's RDF parser from Sesame 0.9.2 against the new
RDF test cases of 25 July 2003[1]. Of the 170 APPROVED parser test
cases, 167 pass and 3 fail.

The 3 failing test cases are:

* rdfms-seq-representation/test001.rdf

    This test fails because Sesame's RDF parser generates
    (_:nodeId, <rdf:type>, <rdf:List>) triples for every bNode that
    is a list. This is conform the specification of 23 January 2003,
    but this has been changed in the editor's draft. The parser will
    be adapted to the new specs once they are officially released.

* rdf-charmod-literals-error001.rdf and
    rdf-charmod-literals-error002.rdf

    This is a known issue in Sesame's RDF parser: it accepts all
    unicode encodings and not just Normal Form C. From Peter F.
    Patel-Schneider's e-mail to rdf-comments [2] we conclude that
    some discussions are going about dropping this constraint from
    RDF. We're waiting for the outcome of this discussion before
    we're starting fixing this issue.

Entailment test cases:
----------------------

We have not yet run a full test run with the Entailment test cases
mentioned in [0]: to be honest the RDF Core WG caught us
off-guard, we were unaware that such a test case set was being
prepared until last friday. Negligence on our part I presume.

Having examined the test cases, we are somewhat surprised that in
fact most test cases seem to deal with borderline details of
datatype entailment, and that there are virtually no tests
whatsoever concerning more mainstream entailment issues, like
class- and property subsumption, cycles in hierarchies, domain and
range restrictions, etc.

Furthermore there are several test that require further
clarification: based on the premise and conclusion data sets it is
often confusing to understand why a particular entailment is or is
not valid.

As a point in case: the tests datatypes/test004(a/b) are both
mentioned as positive and negative tests. It is not explained why
this is so in [0].

The results presented here are not automatically tested, we have
examined the test cases manually, tried out those where we were
unsure how Sesame would react, and for the others the results are
our opinion on how Sesame _would_ react.

Of the 17 approved positive tests, Sesame (0.9.2) passes 7, fails
8, and 2 are unknown.

The 8 failed tests are:

   - datatypes/test007a -> test007b
   - datatypes/test004b -> test004c
   - datatypes/test005a -> test005b
   - datatypes/test003a -> test003b
   - datatypes/test003b -> test003a
   - datatypes/test008a -> test008b

     These all deal with related issues of comparing datatype values
     to each other. Since a lot of the work concerning datatype
     entailments seems to be more or less in flux, and a number of
     editorial changes are anticipated, we will not fix these  until
     a new release of the RDF Semantics WD.

- rdfms-seq-representation/empy -> test002
- rdfms-seq-representation/empy -> test004

     These cases demands that an empty graph
     entails that rdf:_1 is a containermembershipproperty, and a
     subproperty of rdfs:member. The Sesame inferencer is currently
     implemented in such a way that  the rdf:_1 (etc.) properties
     are only added to the inference process when a container is
     added to the repository. Thus, the inference is only made
     later.

- xmlsch-02/test001 -> test003

    According to this test a well-formed XML-Schema-datatyped
    literal entails an empty Literal node. We are unclear how this
    follows from the RDF MT.

The 2 unknown tests are those tests that do not present a
"conclusion" document, but merely state FALSE. We are unsure how
to interpret these tests.

Of the 17 approved negative tests, Sesame (0.9.2) passes 11, fails
1, and 3 unknown.

The 3 unknown tests are:

   - datatypes/test004a -> test004b
   - datatypes/test004b -> test004c

     Both these tests are also mentioned as positive tests,
     therefore it is not quite clear what the expected behaviour of
     the system in fact should be.

   - datatypes/test002 -> test002b
     It is simply unclear to us at this moment what aspect of
     entailment this test is supposed to capture.

Remark: for some reason two tests (statement-entailment/test001a
and statement-entailment/test002a) are listed twice. Sesame passes
the tests, but we are unsure how to interpret the double listing.

The 1 failed test is:

   - xmlsch-02/test002 -> test003

     According to the description the supplied literal is not
     well-formed, but it is not explained why. Clarification in the
     documentation of the test case might be useful.

Editorial remarks:

We feel that the current set of entailment test cases
over-emphasizes the importance of datatype reasoning, and that in
many projects and practical applications, the importance of
correct subsumption reasoning is at least as high, if not much
higher. The set of test cases should reflect that.

Second, we feel that the application of the entailment test cases
is severely under-documented.

We trust this overview is of some help, and welcome your feedback
on our first (admittedly somewhat hurried) findings.

Regards,

Jeen Broekstra and Arjohn Kampman

References:
[0]http://www.w3.org/2001/sw/RDFCore/TR/WD-rdf-testcases-20030117/
[1]http://www.w3.org/2000/10/rdf-tests/rdfcore/
[2]http://lists.w3.org/Archives/Public/www-rdf-comments/2003JulSep/0152.html
[3]http://sesame.aidministrator.nl/

-- 
jeen.broekstra@aidministrator.nl
aidministrator nederland b.v.  -  http://www.aidministrator.nl/
prinses julianaplein 14-b, 3817 cs amersfoort, the netherlands
tel. +31-(0)33-4659987   fax. +31-(0)33-4659987

Received on Monday, 4 August 2003 12:52:23 UTC