[ESWC 2015] Call for Challenge: Concept-Level Sentiment Analysis

** apologies for cross-posting **

==== Call for Challenge: Concept-Level Sentiment Analysis ====

Challenge Website: https://github.com/diegoref/ESWC-CLSA


12th Extended Semantic Web Conference (ESWC) 2015
Dates: May 31 - June 4, 2015
Venue: Portoroz, Slovenia
Hashtag: #eswc2015
Feed: @eswc_conf
Site: http://2015.eswc-conferences.org


GENERAL CHAIR
- Fabien Gandon (Inria, Sophia Antipolis, France)


CHALLENGE COORDINATORS
- Elena Cabrio (Inria, Sophia Antipolis, France)
- Milan Stankovic (SEPAGE, Paris, France)


CHALLENGE CHAIRS
- Mauro Dragoni, FBK, (Italy)
- Valentina Presutti, CNR STLAB Laboratory (Italy)
- Diego Reforgiato Recupero, CNR STLAB Laboratory (Italy)


IMPORTANT DATES
* March 3, 2015, 23:59 CET: Submission due
* April 9, 2015, 23:59 CET: Notification of acceptance
* May 31 - June 4, 2015: Challenge days


MOTIVATION AND OBJECTIVES
As the Web rapidly evolves, Web users are evolving with it. In an era
of social connectedness, people are becoming increasingly enthusiastic
about interacting, sharing, and collaborating through social networks,
online communities, blogs, Wikis, and other online collaborative
media. In recent years, this collective intelligence has spread to
many different areas, with particular focus on fields related to
everyday life such as commerce, tourism, education, and health,
causing the size of the Social Web to expand exponentially.

The opportunity to automatically capture the opinions of the general
public about social events, political movements, company strategies,
marketing campaigns, and product preferences has raised growing
interest within the scientific community, leading to many exciting
open challenges, as well as in the business world, due to the
remarkable benefits deriving from marketing prediction. The
distillation of knowledge from such a large amount of unstructured
information is an extremely difficult task, as the contents of today’s
Web are perfectly suitable for human consumption, but remain hardly
accessible to machines. Mining opinions and sentiments from natural
language, involves a deep understanding of most of the explicit and
implicit, regular and irregular, syntactical and semantic rules proper
of a language. Existing approaches mainly rely on parts of text in
which opinions and sentiments are explicitly expressed such as
polarity terms, affect words and their co-occurrence frequencies.
However, opinions and sentiments are often conveyed implicitly through
latent semantics, which make purely syntactical approaches
ineffective. This issue offers a research opportunity and an exciting
challenge to the Semantic Web community. In fact, concept-level
sentiment analysis aims to go beyond a mere word-level analysis of
text and provide novel approaches to opinion mining and sentiment
analysis that allow a more efficient passage from (unstructured)
textual information to (structured) machine-processable data, in
potentially any domain.

Concept-level sentiment analysis focuses on a semantic analysis of
text through the use of web ontologies, semantic resources, or
semantic networks, allowing the identification of opinion data which
with only natural language techniques would be very difficult. By
relying on large semantic knowledge bases, concept-level sentiment
analysis steps away from blind use of keywords and word co-occurrence
count, but rather relies on the implicit features associated with
natural language concepts. Unlike purely syntactical techniques,
concept-based approaches are able to detect also sentiments that are
expressed in a subtle manner, e.g., through the analysis of concepts
that do not explicitly convey any emotion, but which are implicitly
linked to other concepts that do so.

Systems must have a semantics flavor (e.g., by making use of Linked
Data or known semantic networks within their core functionalities) and
authors need to show how the introduction of semantics can be used to
obtain valuable information, functionality or performance. Existing
natural language processing methods or statistical approaches can be
used too as long as the semantics plays a main role within the core
approach (engines based merely on syntax/word-count will be excluded
from the competition).


TARGET AUDIENCE
The Challenge is open to everyone from industry and academia.


TASKS
The Concept-Level Sentiment Analysis Challenge is defined in terms of
different tasks. The first task is elementary whereas the others are
more advanced. In order to be accepted for the challenge, each system
has to deal with the first task.

- Task 1: Polarity Detection
The basic task of the challenge is binary polarity detection. The
proposed semantic opinion-mining engines will be assessed according to
precision, recall and F-measure of detected polarity values (positive
OR negative) for each review of the evaluation dataset. The problem of
subjectivity detection is not addressed within this Challenge, hence
participants can assume that there will be no neutral reviews.

- Task 2: Aspect-Based Sentiment Analysis
The output of this Task will be a set of aspects of the reviewed
product and a binary polarity value associated to each of such
aspects. So, for example, while for the Elementary Task an overall
polarity (positive or negative) is expected for a review about a
mobile phone, this Task requires a set of aspects (such as ‘speaker’,
‘touchscreen’, ‘camera’, etc.) and a polarity value (positive OR
negative) associated with each of such aspects. Engines will be
assessed according to both aspect extraction and aspect polarity
detection using precision, recall and F-measure similarly as performed
during the first Concept-Level Sentiment Analysis Challenge held
during ESWC2014 and re-proposed at SemEval 2015
(http://www.alt.qcri.org/semeval2015/task12/).

- Task 3: Frame entities Identification
The Challenge focuses on sentiment analysis at concept-level. This
means that the proposed engines must work beyond word/syntax level,
hence addressing a concepts/semantics perspective. This task will
evaluate the capability of the proposed systems to identify the
objects involved in a typical opinion frame according to their role:
holders, topics, opinion concepts (i.e. terms referring to highly
polarised concepts). For example, in a sentence such as The mayor is
loved by the people in the city, but he has been criticized by the
state government (taken from Sentiment Analysis and Opinion Mining,
Bing Liu, 2012), an approach should be able to identify that the
people and state government are the opinion holders, is loved and has
been criticized represent the opinion concepts, The mayor identifies a
topic of the opinion. The proposed engines will be evaluated according
to precision, recall and F-measure.


EVALUATION DATASET
Systems will be evaluated against a testing dataset which will be
released after a first-round of evaluation during the Conference.
Participants are recommended to train and/or test their own systems
using the Blitzer Dataset
(http://cs.jhu.edu/%7Emdredze/datasets/sentiment). Precision, recall,
F1-measure for all the tasks will be computed automatically by a tool
that will be available for download so that each participant will be
able assess their methods and make sure the output produced by their
system is in compliance with the input required by the script.


SUBJECTIVE EVALUATION
A subjective evaluation will be performed by the members of the
Advisory Board. For each system, reviewers will give a numerical score
within the range [1-10] and details motivating their choice. The
scores will be given to the following aspects:
    1. Use of common-sense knowledge and semantics;
    2. Computational time;
    3. Graphical interface - including the number of features that is
possible to query, usability of the system, appealing of the user
interface;
    4. Innovative nature of the approach including multi-language capabilities.


OBJECTIVE EVALUATION
For systems that can be tuned with different parameters, please
indicate a range of up to 5 sets of settings. Settings with the best
precision, recall, F-measure will be considered for judgment. The
objective evaluation will be performed according to precision, recall,
and F-measure analysis.


JUDGING AND PRIZES
We propose to award systems based on two criteria judged separately:
    Subjective: the system with the highest average score in items 1-4 above;
    Objective: the system with the highest score in precision, recall
and F-measure analysis.

After the first round of evaluation a list of runners up will be
defined. A certain number of systems with the highest scores within
the subjective evaluation and a number of systems with the highest
scores within the objective evaluation will be the finalists and will
have to present their work in a conference session. The exact number
will depend on the scores they get and on the Conference policy. They
will have a slot of approximately 15 minutes. The judges will be
present and will evaluate again the systems in more detail. The judges
will then meet in private to discuss the entries and to determine the
winners.


HOW TO PARTICIPATE
The following information has to be provided:
* Abstract: no more than 200 words.
* Description: It should contain the details of the system, including
why the system is innovative, how it uses Semantic Web, which features
or functions the system provides, what design choices were made and
what lessons were learned. The description should also summarize how
participants have addressed the evaluation tasks. Papers must be
submitted in PDF format, following the style of the Springer's Lecture
Notes in Computer Science (LNCS) series
(http://www.springer.com/computer/lncs/lncs+authors), and not
exceeding 12 pages in length.
* Web Access: The application can either be accessible via the web or
downloadable. If the application is not publicly accessible, password
must be provided. A short set of instructions on how to use the
application should be provided as well.

Papers are submitted in PDF format via the challenge's EasyChair
submission pages
(https://easychair.org/conferences/?conf=eswcclsa2015).

Please share comments and questions with the challenge mailing list.
The organizers will assist you for any potential issues that could be
raised.


MAILING LIST
We invite the potential participants to subscribe to our mailing list
in order to be kept up to date with the latest news related to the
challenge.
Mailing list web page is http://groupspaces.com/ESWC2015-CLSA. Users
are invited to join it and then post messages using the address
ESWC2015-CLSA@groupspaces.com


PROGRAM COMMITTEE
* Rada Mihalcea, University of North Texas (USA)
* Ping Chen, University of Houston-Downtown (USA)
* Yongzheng Zhang, LinkedIn Inc. (USA)
* Giuseppe Di Fabbrizio, Amazon Inc. (USA)
* Soujanya Poria, Nanyang Technological University (Singapore)
* Yunqing Xia, Tsinghua University (China)
* Rui Xia, Nanjing University of Science and Technology (China)
* Jane Hsu, National Taiwan University (Taiwan)
* Rafal Rzepka, Hokkaido University (Japan)
* Amir Hussain, University of Stirling (UK)
* Alexander Gelbukh, National Polytechnic Institute (Mexico)
* Bjoern Schuller, Technical University of Munich (Germany)
* Amitava Das, Samsung Research India (India)
* Dipankar Das, National Institute of Technology (India)
* Carlo Strapparava, Fondazione Bruno Kessler (Italy)
* Stefano Squartini, Marche Polytechnic University (Italy)
* Cristina Bosco, University of Torino (Italy)
* Paolo Rosso, Technical University of Valencia (Spain)

Received on Friday, 13 February 2015 15:16:51 UTC