RE: Fact ranking game (creation of ground truth)

Hi Tamara,

 

I have registered and had a look.

 

This is an interesting approach.

 

However, the problems with this are:

 

1.       The survey is already very large. Questions are very extensive.
For volunteers to handle multipage multiple choice questions, it is a great
deal of an effort already.

You are asking about Benjamin Franklin, for example: “Benjamin Franklin has
or had birth date 1706-01-16” – well, of course 99% do not remember this. 

Do you expect them to go online and do the research???? Because currently
the only choice is: “high”, “low”, or “nonsense”

I suggest adding a choice “I don't know” J, because otherwise people will
get scary and will simply drop out of the first page, especially those who
would agree to try to answer these questions casually but not ready to spend
hours and hours on this.



2.       If this is a game, there must be a reward. What’s the reward for
people to go through this? What’s the benefit for them to have higher score?
A trip to Paris? A chocolate medal?

I think you need to work further on motivation of volunteers to complete the
survey.

 

Regards,

Serge

 

 

 

From: Bobic, Tamara [mailto:Tamara.Bobic@hpi.de] 
Sent: Wednesday, August 13, 2014 4:58 PM
To: public-ld4lt@w3.org
Subject: Fact ranking game (creation of ground truth)

 

 

[Apologies for cross-posting. Please redistribute within your own group or
among colleagues, thank you!]

In recent years we have experienced an exponential increase of knowledge
available on the web. Having such an extensive amount of information poses a
challenge when trying to identify globally relevant or important facts. In
effort to address this issue, several fact ranking systems have already been
developed. However, the difficulty with current state-of-the-art systems is
that there is no gold standard corpus which could serve as a ground truth
for evaluating their performances.

We have developed a fun and exciting quiz which will help us to tackle this
issue by using the wisdom of the crowd. While you are playing, your inputs
(which are greatly appreciated) will contribute to our scientific
experiment. Together, we will build a first corpus that will help our
scientific community in evaluating different fact ranking strategies!

 

Here you will find our tool <http://s16a.org/fr/>  that is used to rank
facts about ~500 popular entities from Wikipedia. 

You have to register with the tool and then the task will be explained to
you in detail. You might interrupt your rating of the presented facts any
time you like and continue later. To make it a bit more interesting, you
will be able to score points and see your ranking in a highscore list.

 

We would really appreciate your help in this task. Please do also spread the
word. The more participants, the more valid our ground truth will be.

 

Thanks and best regards,

Semantic Technologies Team

--

Hasso-Plattner-Institut für Softwaresystemtechnik GmbH
Prof.-Dr.-Helmert-Str. 2-3
D-14482 Potsdam
Germany
http://www.hpi.de