Re: Evaluating Exploration/Semantic Search UIs (was: Re: freebase parallax: user interface for browsing graphs of data)

On 17 aug 2008, at 23:22, Andreas Harth wrote:

>
> Hi Michiel, all,
>
> Michiel Hildebrand wrote:
>> Designing user interfaces for "new" ways of exploration is indeed  
>> difficult. Evaluating these interfaces is even more difficult.
>
> I agree with you on the difficulty of evaluation new UIs.  Unlinke  
> traditional fields
> such as IR, there is no corpus or evaluation method available for  
> Semantic
> Web data (at least didn't find any).
>
> To help assessing new UIs for Semantic Web data, we've published a  
> medium-size
> corpus at [1] (~25m triples, ~5GB), together with a set of real- 
> world user tasks.
> There's also ratings which can be used for recommendations.

Thanks Andreas,

A good initiative. I will load the data into our tools and see what it  
looks like.

 From the information I get from the website it looks like you have  
been using the data (and other materials) for evaluation. I can't find  
any results though. The link to the technical report is not working.  
In other words what are your experiences with the use of this dataset  
and your evaluation methodology?

I do not completely get the queries. You list three kinds of queries,  
directed search, simple browsing and complex browsing.
- Why this distinction?
- The description of the queries seem a bit limited.
- Do you have a golden standard for the answers to these queries?

> We've choosen books since there is a public domain data available  
> and there
> has been work in the digital library area, so there are existing  
> systems to
> compare to.


...including evaluations?

Michiel

Received on Monday, 18 August 2008 08:30:39 UTC