CfP: EvalUMAP 2017 Workshop. Towards comparative evaluation in user modeling ,adaptation and personalization (EvalUMAP 2016)

*


  Call for Papers

Submission deadline: April 20, 2017

*To be held in conjunction with the 25th Conference on User Modeling, 
Adaptation and Personalization, UMAP 2017, July 2017, Bratislava, Slovakia*


Research in the areas of User Modelling, Adaptation and Personalization 
faces a number of significant scientific challenges. One of the most 
significant of these challenges is the issue of comparative evaluation. 
It has always been difficult to rigorously compare different approaches 
to personalization, as the function of the resulting systems is, by 
their nature, heavily influenced by the behaviour of the users involved 
in trialling the systems. To-date this topic has received relatively 
little attention. Developing comparative evaluations in this space would 
be a huge advancement as it would enable shared comparison across 
research, which to-date has been very limited.

Taking inspiration from communities, such as Information Retrieval and 
Machine Translation, the EvalUMAP Workshop series seek to propose and 
design one or more shared tasks to support the comparative evaluation of 
approaches to User Modelling, Adaptation and Personalization. This 
year’s workshop will solicit presentations from key practitioners in the 
field on innovative datasets that meet specific requirements (e.g. 
ownership, accessibility, privacy) and that could form the basis to 
start scoping and designing shared task-based challenges and evaluations 
in the area of user adaptation and personalisation for next year. The 
resulting shared task(s) will be accompanied by appropriate models, 
content, metadata, user behaviours, etc., and can be used to 
comprehensively compare how different approaches and systems perform. In 
addition, a number of evaluation metrics and methods will be outlined, 
that participants would be expected to perform in order to facilitate 
comparison. Finally, the proposed shared task(s) will be disseminated in 
the community and the resulting outcomes will be presented at an 
EvalUMAP forum next year.


In particular, the planned outcomes of the EvalUMAP Workshop 2017 is as 
follows: (1) A clear understanding of the challenges and requirements 
related to the design of a shared task-based approach in User Modeling, 
Adaptation and Personalization space and (2) identify specific issues 
and requirements on user data capturing and dataset processing in the 
context of personalization for a shared task (3) the identification and 
description of suitable and publicly accessible datasets that overcome 
the previous identified challenges and (4) the design of shared 
task-based evaluations using suitable datasets that will take place 
throughout 2017 and early 2018, and be presented at UMAP 2018.

Workshop topics are evaluation focused and include, but are not limited to:

  *

    Understanding UMAP evaluation

  *

    Defining tasks and scenarios for evaluation purposes

  *

    Identification of potential corpora (datasets) for shared tasks

  *

    Automated and semi automated processes on creating appropriate
    datasets and simulating user behaviours etc in order to accommodate
    a shared task

  *

    Interesting target tasks and explanations of their importance

  *

    Critiques or comparisons of existing evaluation metrics and methods

  *

    How we can combine existing evaluation metrics and methods?

  *

    Reusing or improving previously suggested metrics and methods

  *

    Reducing the cost of evaluation

  *

    Proposal of new evaluation metrics and methods

  *

    Technical challenges associated with design and implementation

  *

    Anonymization of datasets, Privacy, Ethics and Security issues on
    the use of datasets


Workshop format:


This will be an interactive workshop structured to encourage group 
discussion and active collaboration among attendees. The workshop will 
feature a keynote talk, lightning round presentation session for 
position papers, multiple (parallel) breakout sessions, and a final 
discussion session to wrap up the event.

Paper Submissions

The workshop is now accepting position papers from 2 to 6 pages 
(including references) describing approaches or ideas/challenges on the 
topics of the workshop are invited. In particular, three types of papers 
are solicited:  (1) papers describing available infrastructure that 
could be used to capture data for shared evaluation challenges; (2) 
  papers describing available datasets that could be exploited for 
shared challenge generation; (3) papers describing challenges and 
potential solutions around shared challenge generation in the UMAP space.


Submissions should be in ACM Standard SIGCONF format. LaTeX and Word 
templates are available 
at<http://www.acm.org/sigs/publications/proceedings-templates>http://www.acm.org/publications/proceedings­-template 
<http://www.acm.org/publications/proceedings-template>.

Papers should be submitted in pdf format through the EasyChair 
systemhttps://easychair.org/conferences/?conf=evalumap2017no later than 
midnight 11:59pm Hawaii time on April 20, 2017. Submissions will be 
reviewed by members of the workshop program committee. Accepted papers 
will be included in the extended UMAP 2017 Proceedings and will be 
available via the ACM Digital Library. In addition, the EvalUMAP 
workshop proceedings will be indexed with CEUR. Authors of select papers 
may be invited to contribute to a journal publication which describes 
the outcomes of the workshop.

Important Dates

April 20, 2017: Deadline for paper submission (11:59pm Hawaii time)

May 20, 2017: Notification to authors

May 28, 2017: Camera-ready paper due

July 9, 2017: EvalUMAP Workshop at UMAP

Further Information

Further information is available on the workshop website 
athttp://evalumap.adaptcentre.ie/or by emailing the workshop organizers 
at evalumap@adaptcentre.ie.

Workshop Organizers

Owen Conlan, Trinity College Dublin, Ireland

Liadh Kelly, Dublin City University, Ireland

Kevin Koidl, Trinity College Dublin, Ireland

Séamus Lawless, Trinity College Dublin, Ireland

Athanasios Staikopoulos, Trinity College Dublin, Ireland


Programme Committee

Paul De Bra, Eindhoven University of Technology, The NetherlandsIván 
Cantador, Universidad Autónoma de Madrid, SpainDavid Chin, University of 
Hawaii, USA

William Wright, University of Hawaii, USA

Eelco Herder, L3S Research Center, Hannover, GermanyGeert-Jan Houben, 
Delft University of Technology, The NetherlandsJudy Kay, University of 
Sydney, AustraliaTsvi Kuflik, The University of Haifa, IsraelAlexandros 
Paramythis, Contexity, SwitzerlandFrancesco Ricci, University of Rome, 
ItalyAlan Said, Recorded Future, SwedenVincent Wade, Trinity College 
Dublin, IrelandStephan Weibelzahl, Private University of Applied 
Sciences Göttingen, Germany*