EVALITA 2023: Call for tasks - NEW DEADLINES and TIMELINE

*

*****************************************************************************

EVALITA 2023: Call for tasks - NEW DEADLINES and TIMELINE

*****************************************************************************


EVALITA 2023is an initiative of AILC(Associazione Italiana di 
Linguistica Computazionale, AILC https://www.ai-lc.it/ 
<https://www.ai-lc.it/>). The final workshop will be held on **September 
7th-8th in Parma, Italy**.


As in the previous editions (https://www.evalita.it/ 
<https://www.evalita.it/>), EVALITA 2023 will be organized along a few 
selected tasks, which provide participants with opportunities to discuss 
and explore both emerging and traditional areas of Natural Language 
Processing and Speech for Italian. The participation is encouraged for 
teams working both in

academic institutions and industrial organizations.


TASK PROPOSAL SUBMISSION

Tasks proposals should be no longer than 4 pages and should include:

- task title and acronym;

- names and affiliation of the organizers (minimum 2 organizers);

- brief task description, including motivations and state of the art;

- explanation of the international relevance of the task;

- description and examples of the data, including information about 
their availability, development stage, and issues concerning privacy and 
data sensitivity. The examples are mandatory because they are intended 
to give potential participants an idea of what the task data will look 
like, how it’ll be formatted, etc.

- expected number of participants and attendees;

- names and contact information of the organizers.


In submitting your proposal, please bear in mind that we encourage:

- challenging tasks involving linguistic analysis, e.g., beyond “simple” 
classification problems;

- tasks focused on multimodality, e.g., considering both textual and 
visual information;

- tasks characterized by different levels of complexity, e.g., with a 
straightforward main subtask and one or more sophisticated additional 
subtasks;

- the re-annotation/expansion of datasets from previous years with new 
annotation levels, and texts from publicly available corpora;

- both new tasks and re-runs: for new tasks, organizers will have to 
specify in the proposal why it would attract a reasonable number of 
participants, and why it is needed;

- application-oriented tasks, that is tasks that have a clearly defined 
end-user application showcasing;

- multilingual tasks, i.e. with data both in Italian and in other languages;

- industrial tasks, i.e. tasks with real data provided by companies.



The organizers of the accepted tasks should take care of planning, 
according to the scheduled deadlines (see below):

- the development and distribution of datasets needed for the contest, 
i.e. data for training and development, and data for testing; the scorer 
to be used to evaluate the submitted systems should be included in the 
release of development data;

- the development of task guidelines, where all the instructions for the 
participation are made clear together with a detailed description of 
data and evaluation metrics applied for the evaluation of the 
participant results;

- the collection of participants results;

- the evaluation of participants results according to standard metrics 
and baseline(s);

- the solicitation of participation and of submissions;

- the reviewing process of the papers describing the participants 
approach and results (according to the template to be made available by 
the EVALITA 2023 chairs);

- the production of a paper describing the task (according to the 
template to be made available by the EVALITA 2023 chairs).



*** Email your proposal in PDF format to evalita2023@gmail.com with 
"EVALITA 2023 TASK Proposal" as the subject line by the submission 
deadline: October 17th 2022. ***


Please feel free to contact the EVALITA 2023 chairs at 
evalita2023@gmail.com in case of any questions or suggestions.


NEW deadlines of the task proposal:

- October 4th 2022 October 17th 2022: submission of task proposals

- October 18th 2022 October 24th 2022: notification of task proposal 
acceptance


NEW timeline of EVALITA 2023:

- 7th February 2023: development data available to participants

- 30th April 2023: registration closes

- 2nd-19th May 2023: evaluation windows

- 30th May 2023: assessment returned to participants

- 14th June 2023: final reports (from participants) due to task organizers

- 28th June 2023: final reports (from task organizers) due to EVALITA chairs

- 10th July 2023: review deadline

- 25th July 2023: camera ready version deadline

- 7th-8th September 2023: final workshop in Parma



EVALITA 2023 CHAIRS

Mirko Lai (Università di Torino)

Stefano Menini (Fondazione Bruno Kessler)

Marco Polignano (Università di Bari Aldo Moro)

Valentina Russo (Logogramma SRL)

Rachele Sprugnoli (Università degli Studi di Parma)

Giulia Venturi (Istituto di Linguistica Computazionale “A. Zampolli” - CNR)

*