W3C home > Mailing lists > Public > semantic-web@w3.org > August 2008

Re: Evaluating Exploration/Semantic Search UIs (was: Re: freebase parallax: user interface for browsing graphs of data)

From: Michiel Hildebrand <Michiel.Hildebrand@cwi.nl>
Date: Mon, 18 Aug 2008 10:30:24 +0200
To: Andreas Harth <andreas.harth@deri.org>
Message-Id: <4977F404-6784-4631-8C70-0165CA9703B8@cwi.nl>
Cc: semantic-web@w3c.org

On 17 aug 2008, at 23:22, Andreas Harth wrote:

> Hi Michiel, all,
> Michiel Hildebrand wrote:
>> Designing user interfaces for "new" ways of exploration is indeed  
>> difficult. Evaluating these interfaces is even more difficult.
> I agree with you on the difficulty of evaluation new UIs.  Unlinke  
> traditional fields
> such as IR, there is no corpus or evaluation method available for  
> Semantic
> Web data (at least didn't find any).
> To help assessing new UIs for Semantic Web data, we've published a  
> medium-size
> corpus at [1] (~25m triples, ~5GB), together with a set of real- 
> world user tasks.
> There's also ratings which can be used for recommendations.

Thanks Andreas,

A good initiative. I will load the data into our tools and see what it  
looks like.

 From the information I get from the website it looks like you have  
been using the data (and other materials) for evaluation. I can't find  
any results though. The link to the technical report is not working.  
In other words what are your experiences with the use of this dataset  
and your evaluation methodology?

I do not completely get the queries. You list three kinds of queries,  
directed search, simple browsing and complex browsing.
- Why this distinction?
- The description of the queries seem a bit limited.
- Do you have a golden standard for the answers to these queries?

> We've choosen books since there is a public domain data available  
> and there
> has been work in the digital library area, so there are existing  
> systems to
> compare to.

...including evaluations?

Received on Monday, 18 August 2008 08:30:39 UTC

This archive was generated by hypermail 2.4.0 : Tuesday, 5 July 2022 08:45:08 UTC