- From: mlw05r <mlw05r@ecs.soton.ac.uk>
- Date: Mon, 18 Aug 2008 10:58:27 +0100
- To: Michiel Hildebrand <Michiel.Hildebrand@cwi.nl>
- Cc: Andreas Harth <andreas.harth@deri.org>, semantic-web@w3c.org
there was an interesting issue of IP&M about evaluating more complex exploration behaviours [1]. i would recommend it. there was also a CACM special issue on what is exploratory search behaviour a couple of years ago [2]. it's all almost completely outside of semantic web specific work. worth looking at tho, if you haven't already, as i am sure the questions will apply. certainly the SW is opening up new exploratory interfaces, as well as improving 'known item' search. btw. Marchionini and his team at UNC defined 4: simple search (one facet used), complex search (multiple facets of information required), directed browsing (as in learning with a goal in mind) and undirected browsing (browsing, but not knowing what the aim is yet). This was in their JCDL07 paper [3], i think. If its not in there, they certainly disscussed them while designing the study. They had some really interesting notions that things like 'time' are not good evaluation measures for the browsing tasks, as potentially the aim is to learn as much as possible, rather than get an answer quickly. thus any notion of stopping after a short amount of time might mean that they have not found a lot of information on a topic. My own work [4] has looked at how you can, through systematic inspection, estimate (without running user studies) the range of strategies and tactics users can employ with a UI. combined with how much support each tactic receives, you can get, for free, the amount that different types of users are supported (where user types are based on goal, ability to specify, etc). Max Wilson Univesrity of Southampton, UK http://users.ecs.soton.ac.uk/mlw05r/ [1] http://www.sciencedirect.com/science/journal/03064573 [2] http://doi.acm.org/10.1145/1121949.1121978 (introductory article - you can get the table of contents for the issue from there) [3] http://doi.acm.org/10.1145/1255175.1255267 [4] http://eprints.ecs.soton.ac.uk/14301/ (pre-revisions version of an article). my latest tech report for the phd has more detail, but is also much bigger (http://eprints.ecs.soton.ac.uk/15642/) On Mon, 18 Aug 2008 10:30:24 +0200, Michiel Hildebrand <Michiel.Hildebrand@cwi.nl> wrote: > > > On 17 aug 2008, at 23:22, Andreas Harth wrote: > >> >> Hi Michiel, all, >> >> Michiel Hildebrand wrote: >>> Designing user interfaces for "new" ways of exploration is indeed >>> difficult. Evaluating these interfaces is even more difficult. >> >> I agree with you on the difficulty of evaluation new UIs. Unlinke >> traditional fields >> such as IR, there is no corpus or evaluation method available for >> Semantic >> Web data (at least didn't find any). >> >> To help assessing new UIs for Semantic Web data, we've published a >> medium-size >> corpus at [1] (~25m triples, ~5GB), together with a set of real- >> world user tasks. >> There's also ratings which can be used for recommendations. > > Thanks Andreas, > > A good initiative. I will load the data into our tools and see what it > looks like. > > From the information I get from the website it looks like you have > been using the data (and other materials) for evaluation. I can't find > any results though. The link to the technical report is not working. > In other words what are your experiences with the use of this dataset > and your evaluation methodology? > > I do not completely get the queries. You list three kinds of queries, > directed search, simple browsing and complex browsing. > - Why this distinction? > - The description of the queries seem a bit limited. > - Do you have a golden standard for the answers to these queries? > >> We've choosen books since there is a public domain data available >> and there >> has been work in the digital library area, so there are existing >> systems to >> compare to. > > > ...including evaluations? > > Michiel
Received on Monday, 18 August 2008 10:04:43 UTC