Semantic Reasoning Evaluation Challenge (SemREC) at ISWC 2021

Dear All,

TL;DR
*************************
We are organizing a challenge centered around reasoning. We invite you to
make submissions that can be in one or more of the following categories -
1) an ontology developed for a real-world application, but proved to be a
challenge for the existing reasoners, 2) A traditional description logic
reasoner that was developed or made improvements to in the last few years,
3) A neuro-symbolic reasoner that approximates entailments or predicts
missing axioms. If you have any questions, please contact us (email IDs are
given in the cc).

Further details are on the website: https://semrec.github.io/
*************************


Longer version
*************************

Despite the development of several ontology reasoning optimizations, the
traditional methods either do not scale well or only cover a subset of OWL
2 language constructs. As an alternative, neuro-symbolic approaches are
gaining significant attention. However, the existing methods still can not
deal with very expressive ontology languages. To find and improve these
performance bottlenecks of the reasoners, we ideally need several
real-world ontologies that span the broad spectrum in terms of their size
and expressivity. However, that is often not the case. One of the potential
reasons for the ontology developers to not build ontologies that vary in
terms of size and expressivity is the performance bottleneck of the
reasoners. This challenge includes three tasks that aim to deal with this
chicken and egg problem.


Task-1 - Submit a real-world ontology that is a challenge in terms of the
reasoning time or memory consumed during reasoning. We will be evaluating
the submitted ontologies based on the time and the memory consumed for a
reasoning task, such as classification.

Task-2 - Submit a description logic reasoner that makes use of traditional
techniques such as tableau algorithms and saturation rules. We will
evaluate the performance and the scalability of the submitted systems on
the datasets based on the time taken and memory consumed on the ontology
classification task. This will provide an insight into the progress in the
development of reasoners since the last reasoner evaluation challenge (ORE
2015).

Task-3 - Submit an ontology/RDFS reasoner that makes use of neuro-symbolic
techniques for reasoning and optimization. We will be evaluating two types
of neuro-symbolic systems: (a) that approximate the entailment reasoning
for addressing the time complexity problem, or (b) predicting missing and
plausible axioms for completion. We will evaluate the submitted systems on
the test datasets based on the time taken, memory consumed, precision and
recall.


Submission Details

Participants are requested to make a manuscript submission describing their
entry. For Task 1, we expect a detailed description of the ontology along
with the analysis of the reasoning performance, the workarounds, if any,
that were used to make the ontology less challenging (for example, dropping
of a few axioms, redesigning the ontology, etc.), and the (potential)
applications in which the ontology could be used. For Tasks 2 and 3, we
expect a detailed description of the system, including evaluating the
system on the provided datasets. The submissions can be either in the form
of short papers of length 5 pages or long papers of length 10-12 pages. All
the submissions must be in English and follow the 1-column CEUR-ART style
(link to the overleaf page would be updated soon). The proceedings will be
published as a volume of CEUR-WS. Submissions should be made in the form of
a pdf document on EasyChair (link will be updated soon).

Website: https://semrec.github.io/

Organizers

Gunjan Singh, IIIT-Delhi, India.
Raghava Mutharaju, IIIT-Delhi, India.
Pavan Kapanipathi, IBM T.J. Watson Research Center, USA.