W3C home > Mailing lists > Public > public-rdf-comments@w3.org > December 2013

Re: RDF 1.1 Semantics Implementation Report on the Swertia RDF-Based Reasoner

From: Gregg Kellogg <gregg@greggkellogg.net>
Date: Sun, 8 Dec 2013 16:09:52 -0800
Cc: "public-rdf-comments@w3.org Comments" <public-rdf-comments@w3.org>, "Peter F. Patel-Schneider" <pfpschneider@gmail.com>
Message-Id: <07D2B53D-4773-4A38-AE7E-029FE4C2609E@greggkellogg.net>
To: Michael Schneider <schneid@fzi.de>
On Dec 8, 2013, at 12:54 PM, Michael Schneider <schneid@fzi.de> wrote:

> Dear Working Group!
> Today, I wanted to do all the testing for my reasoner
> and was ready to spend half the day with this activity.
> Unfortunately, I found that this is not as easy as I
> thought.
> The procedure would be to first parse a "manifest.ttl" file
> into an RDF graph, get all the metadata and links to test
> files from it, find out the sort of test case
> (positive/negative entailment/satisfiability test),
> call the reasoner with pointers to the files from the
> manifest, telling the file type (yes, the test files
> really exist in different serialization formats!),
> collecting the result value, of course, checking for
> processing errors or timeouts, and finally building
> up the singular EARL RDF graphs and the complete result
> graph with the project-specific metadata.
> Now, that's a lot of stuff to implement (and to understand
> in the first place), and most of it is totally generic for
> any reasoning tool used with the RDF 1.1 test suite,
> and much of it, such as the particular manifest file or EARL
> format, should not even be of interest to the tool provider.
> What I would like to have is a tool that does all the testing
> and just calls the reasoner with a defined input/output
> behaviour, such that all I have to implement additionally
> is a thin wrapper around my reasoner that handles this
> input/output behaviour.
> Is there such a tool, and I have only missed this?

I don't know of a standard tool to do precisely what you're asking for, but I implement a common pattern among my RDF serialization testers. For example, in <https://github.com/ruby-rdf/rdf-turtle>, where I have spec/suite_helper.rb and spec/suite_spec.rb, which parse the manifest and provide helpers for each test there. Extending this to work for the semantics suite should be fairly straight-forward, if you have some familiarity with Ruby. I suspect that other systems (such as Jena) have similar infrastructures for running tests.

There's also a similar tool for the RDFa test suite: <https://github.com/rdfa/rdfa-website/>, which provides a wrapper to call out to different executables to perform an RDFa transformation and tests the results (see bin/run_suite).

Performing timeouts was beyond the scope of these tools, but would be straight-forward.

If any of these can be of help to you, you're welcome to them; all my code is public-domain.


> The proposed I/O protocol for the reasoner (wrapped-into-a-command-line-tool) would be:
>  Input parameters:
>  * input file name 1
>  * input file serialization 1
>  * input file name 2 (only for entailment checks)
>  * input file serialization 2 (only for entailment checks)
>  Output (on stdout): TRUE | FALSE | ERROR
> The testing tool itself would be responsible for checking for
> timeouts and unhandled errors (other than those indicated
> by "ERROR" by the tool itself), and would be called with
> following input parameters:
>  * project metadata file name (see below)
>  * testdata folder name (which includes the "manifest.ttl" file)
>  * output file name
>  * timeout (in seconds)
>  * reasoner file name (the executable)
> The project metadata file would have to be created by the
> reasoner provider and would contain the fixed metadata
> from which the result EARL file is being created, both the
> metadata for the whole test run as well as the fixed metadata
> for each singular test result. This metadata includes
> the project URI, developer, etc. The appropriate format
> may be RDF or a key=value format, or whatever.
> If no such tool exists or is being created by the WG,
> I have to do this myself, which will delay things further,
> I'm afraid.
> Regards,
> Michael
> Am 04.12.2013 18:25, schrieb Peter F. Patel-Schneider:
>> Hi Michael
>> The RDF 1.1 entailment tests have been changed a bit, to conform better
>> to what RDF systems to.
>> When you run the tests, could you make sure that you have an up to date
>> set?
>> Thanks,
>> peter
>> On 12/02/2013 03:20 PM, Michael Schneider wrote:
>>> Dear Working Group,
>>> please find below my implementation report for my experimental Swertia
>>> RDF-Based Reasoner, a system that tries to be a close implementation
>>> of the model-theoretic semantics of RDF (unlike the many existing
>>> systems that are more based on the RDF entailment rules). I still
>>> wasn't able to run the official RDF 1.1 tests, due to lack of time. I
>>> also believe that the result for the test suite will not become very
>>> good, as many of the tests are about datatype reasoning, which is not
>>> supported by my system. Anyway, I still plan to run the tests, as soon
>>> I find the time, and also plan to provide the results and the
>>> prototypical system, but for now I provide you with my implementation
>>> experiences only. I hope this will already be useful for the Working
>>> Group.
>>> Best regards,
>>> Michael
Received on Monday, 9 December 2013 00:10:23 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 16:59:44 UTC