[ESWC 2015] - Call for Challenges Proposal

=============================================================
ESWC 2015 - 12th Extended Semantic Web Conference 2015
Call for Semantic Web Challenges Proposals
=============================================================

After last year’s success, ESWC organizers are glad to announce that
the Challenges Track will be included again in the program of ESWC
2015! Three challenges were held last year [1] and allowed the
conference to attract a broader audience spanning across disciplines
related to Semantic Web (such as recommender systems). This year, a
call for challenges is open in order to open the selection of
challenges to be held at the conference.

The purpose of challenges is to showcase the maturity of the state of
the art on tasks common to Semantic Web community and adjacent
academic communities, in a controlled setting of rigorous evaluation.

Semantic Web Challenges are an official track of the conference, and
the challenge participants, must provide, in addition to their
participation to the challenge, a paper describing their approach.
This paper must undergo a peer-review by experts relevant to the
challenge task, and will be published in the official proceedings.

Challenge organizers are encouraged to submit proposals with the
following qualities:

- More than one task. Tasks should be independent (participants can
participate in one or more). All tasks should be related to Semantic
Web. Semantic Web technologies should play a prominent role.

- Task descriptions correspond to the interests of a wider audience.
We encourage the challenge organizers to propose at least one basic
task that can be addressed by a larger audience from their community.
Obtaining comments from potential future participants can also help
drive more contenders to the challenge.

- Clear and rigorous definition of the tasks. For each task, you
should define a deterministic and objective way to verify if the goal
of the task has been achieved, and to which extent it has been
achieved (if applicable). The best way is usually to provide detailed
examples of input data and expected output. The examples must cover
all the possible situations that can occur while performing the task,
and should leave no place to ambiguity about whether in a particular
case the task is done or not.

- Valid dataset (if applicable). You should find or create a dataset
that will be used for the challenge. In any case, you must specify the
provenance of the dataset (if it contains human annotation – how were
those obtained). You must make sure you have the right to use/publish
this dataset and clearly state the license for its use within the
challenge. The dataset should be split in two parts – the training
part, and the evaluation part. The training part contains the data,
and the results that should be obtained when performing the task. As
for the evaluation part, you should only publish the data, and make
sure that the correct results have not previously been available to
the participants. When proposing the challenge you must provide
details on the dataset and on the way it is/will be created – the
dataset can be made available later.

- Challenge Committee: Composed of at least 3 respected researchers
with experience in the tasks of the challenge. They help evaluate the
papers submitted by the participants, and also validate the evaluation
procedure.

- Evaluation metrics and procedure. For each task there must be a
number of objective criteria (metrics), e.g. precision and recall. The
evaluation procedure and the way in which the metrics will be
calculated must be clearly specified and made transparent to
participants.

Selection criteria:
-       Relevance for the Semantic Web community
-       Potential number to interested participants
-       Rigor and transparency of the evaluation procedure
-       Endorsements (from researchers working on the task, from
industry players interested in results, from future participants)

IMPORTANT DATES

Challenges proposals due Friday November 21, 2014 - 23:59 Hawaii Time
Challenges chosen/merged – notification to organizers sent Friday
December 5, 2014
Training data ready and challenges Calls for Papers sent January 15th, 2015
Challenge papers submission deadline – March 3rd, 2015
Challenge paper reviews due – April 5th, 2015
Notifications sent to participants and invitations to submit task
results – April 9th, 2015
Test data (and other participation tools) published – April 9th, 2015
Camera ready papers due - April 24th, 2015
Submission of challenge results – free choice of organizers
Proclamation of winners – During ESWC closing Ceremony

SUBMISSION DETAILS

The challenges proposals should contain at least the following elements:

- A summary description of the challenge and tasks
- How the training/testing data will be built and/or procured
- The evaluation methodology to be used, including clear evaluation
criteria and the exact way in which they will be measured. Who will
perform the evaluation and how will transparency be assured?
- The anticipated availability of the necessary resources to the participants
- The resources required to prepare the tasks (computation and
annotation time, costs of annotations, etc)
- The list of challenge committee members who will evaluate the
challenge papers (please indicate which of the listed members already
accepted the role)

In case of doubt, feel free to send us your challenge proposal drafts
as early as possible – the challenges chairs will provide you with
feedback and answers to questions you may have.

Please submit proposals as soon as possible, preferably by electronic
mail in PDF format to the ESWC Challenges chairs:

Milan Stankovic, Sépage S.A.S., France & Université Paris-Sorbonne
(milstan@sepage.fr)
Elena Cabrio, INRIA Sophia Antipolis, France (elena.cabrio@inria.fr)


[1] http://2014.eswc-conferences.org/important-dates/call-challenges

Received on Monday, 17 November 2014 14:30:05 UTC