W3C home > Mailing lists > Public > semantic-web@w3.org > August 2008

Re: Evaluating Exploration/Semantic Search UIs (was: Re: freebase parallax: user interface for browsing graphs of data)

From: mlw05r <mlw05r@ecs.soton.ac.uk>
Date: Mon, 18 Aug 2008 10:58:27 +0100
To: Michiel Hildebrand <Michiel.Hildebrand@cwi.nl>
Cc: Andreas Harth <andreas.harth@deri.org>, semantic-web@w3c.org
Message-ID: <a939f3199eb3fdfe7b950e0db949be9e@ecs.soton.ac.uk>

there was an interesting issue of IP&M about evaluating more complex
exploration behaviours [1]. i would recommend it. there was also a CACM
special issue on what is exploratory search behaviour a couple of years ago
[2]. it's all almost completely outside of semantic web specific work.
worth looking at tho, if you haven't already, as i am sure the questions
will apply. certainly the SW is opening up new exploratory interfaces, as
well as improving 'known item' search.

btw. Marchionini and his team at UNC defined 4: simple search (one facet
used), complex search (multiple facets of information required), directed
browsing (as in learning with a goal in mind) and undirected browsing
(browsing, but not knowing what the aim is yet). This was in their JCDL07
paper [3], i think. If its not in there, they certainly disscussed them
while designing the study. They had some really interesting notions that
things like 'time' are not good evaluation measures for the browsing tasks,
as potentially the aim is to learn as much as possible, rather than get an
answer quickly. thus any notion of stopping after a short amount of time
might mean that they have not found a lot of information on a topic.

My own work [4] has looked at how you can, through systematic inspection,
estimate (without running user studies) the range of strategies and tactics
users can employ with a UI. combined with how much support each tactic
receives, you can get, for free, the amount that different types of users
are supported (where user types are based on goal, ability to specify,
etc). 

Max Wilson
Univesrity of Southampton, UK
http://users.ecs.soton.ac.uk/mlw05r/

[1] http://www.sciencedirect.com/science/journal/03064573
[2] http://doi.acm.org/10.1145/1121949.1121978 (introductory article - you
can get the table of contents for the issue from there)
[3] http://doi.acm.org/10.1145/1255175.1255267
[4] http://eprints.ecs.soton.ac.uk/14301/ (pre-revisions version of an
article). my latest tech report for the phd has more detail, but is also
much bigger (http://eprints.ecs.soton.ac.uk/15642/)

On Mon, 18 Aug 2008 10:30:24 +0200, Michiel Hildebrand
<Michiel.Hildebrand@cwi.nl> wrote:
> 
> 
> On 17 aug 2008, at 23:22, Andreas Harth wrote:
> 
>>
>> Hi Michiel, all,
>>
>> Michiel Hildebrand wrote:
>>> Designing user interfaces for "new" ways of exploration is indeed
>>> difficult. Evaluating these interfaces is even more difficult.
>>
>> I agree with you on the difficulty of evaluation new UIs.  Unlinke
>> traditional fields
>> such as IR, there is no corpus or evaluation method available for
>> Semantic
>> Web data (at least didn't find any).
>>
>> To help assessing new UIs for Semantic Web data, we've published a
>> medium-size
>> corpus at [1] (~25m triples, ~5GB), together with a set of real-
>> world user tasks.
>> There's also ratings which can be used for recommendations.
> 
> Thanks Andreas,
> 
> A good initiative. I will load the data into our tools and see what it
> looks like.
> 
>  From the information I get from the website it looks like you have
> been using the data (and other materials) for evaluation. I can't find
> any results though. The link to the technical report is not working.
> In other words what are your experiences with the use of this dataset
> and your evaluation methodology?
> 
> I do not completely get the queries. You list three kinds of queries,
> directed search, simple browsing and complex browsing.
> - Why this distinction?
> - The description of the queries seem a bit limited.
> - Do you have a golden standard for the answers to these queries?
> 
>> We've choosen books since there is a public domain data available
>> and there
>> has been work in the digital library area, so there are existing
>> systems to
>> compare to.
> 
> 
> ...including evaluations?
> 
> Michiel
Received on Monday, 18 August 2008 10:04:43 GMT

This archive was generated by hypermail 2.3.1 : Tuesday, 26 March 2013 21:45:24 GMT