EVALITA 2016: Call for Tasks

(Apologies for cross-posting)

*** EVALITA 2016: CALL for TASKS ***


The Submission deadline is Sunday, 21 February 2016.


The 5th evaluation campaign EVALITA 2016 will be held in Naples, as a 
satellite event of the 3rd Italian Conference on Computational 
Linguistics - CLiC-it 2016 (December 5-7, 2016).

EVALITA 2016 is an initiative of AILC (Associazione Italiana di 
Linguistica Computazionale, http://www.ai-lc.it/).


As in the previous editions (http://www.evalita.it/), EVALITA 2016 will 
be organized along few selected tasks, which provide participants with 
opportunities to discuss and explore both emerging and traditional areas 
of Natural Language Processing and Speech for Italian. The participation 
is encouraged for teams working both in academic institutions and 
industrial organizations.


TASK PROPOSAL SUBMISSION


Tasks proposals should be no longer than 4 pages and should include:

  *

    task title and acronym

  *

    names and affiliation of the organizers (minimum 2 organizers)

  *

    brief task description, including motivations and state of the art
    showing the international relevance of the task

  *

    description and examples of the data needed for the task, including
    information about their availability and development stage

  *

    expected number of participants and attendees

  *

    names and contact information of the organizers


In submitting your proposal, please bear in mind that we encourage:

  *

    the re-annotation of datasets from previous years with new
    annotation types or texts from publicly available corpora

  *

    application-oriented tasks, that is tasks that have a clearly
    defined end-user application showcasing

  *

    both new tasks and re-runs. For new tasks, you will have to specify
    in the proposal why this would attract a reasonable number of
    participants, and why is needed. For a re-run, you should defend in
    your proposal the need for having such task again (new metrics, new
    types of data, new domains, growing interest, etc.)


The organizers of the accepted tasks should take care of planning, 
according to the scheduled deadlines (see below):

  *

    the development and distribution of data sets needed for the
    contest, i.e. data for training and development, and data for
    testing; the scorer to be used to evaluate the submitted systems
    should be included in the release of development data

  *

    the development of task guidelines, where all the instructions for
    the participation are made clear together with a detailed
    description of data and evaluation metrics applied for the
    evaluation of the participant results; the collection of
    participants results

  *

    the evaluation of participants results according to standard metrics

  *

    the solicitation of submissions

  *

    the reviewing process of the papers describing the participants
    approach and results (according to the template to be made available
    by the EVALITA-2016 chairs)

  *

    the production of a paper describing the task (according to the
    template to be made available by the EVALITA-2016 chairs)


Email your proposal in PDF format to evalita2016@gmail.com 
<mailto:evalita2016@gmail.com>with "Evalita 2016 TASK Proposal" as the 
subject line by the submission deadline (21st February 2016).

Please feel free to contact the EVALITA-2016 chairs at 
evalita2016@gmail.com <mailto:evalita2016@gmail.com>in case of any 
questions or suggestions.


IMPORTANT DATES:


- 21th February 2016: submission of task proposals

- 7th March 2016: notification of task proposal acceptance; call for 
Interest opens

- 14th March 2016: on-line registration opens

- 30th May 2016: development data available to participants

- 12th September 2016: test data available, registration closes

- 19th September 2016: system results due to organizers

- 26th September 2016: assessment returned to participants

- 31st October 2016: technical reports due to organizers

- 5th-7th December 2016: final workshop (one day during CLIC-2016)


EVALITA-2016 CHAIRS

Pierpaolo Basile (Università degli Studi di Bari Aldo Moro)

Franco Cutugno (Università degli Studi di Napoli Federico II)

Malvina Nissim (Center for Language and Cognition Groningen, 
Rijksuniversiteit Groningen)

Viviana Patti (Università degli Studi di Torino)

Rachele Sprugnoli (Digital Humanities Group, Fondazione Bruno Kessler)


CONTACT:

evalita2016@gmail.com <mailto:evalita2016@gmail.com>