R: R: sws matchmaker contest

Dear all,
I found the discussion so far definitely interesting. 

I believe Matthias idea for a contest is timely and useful for the whole
community. We need to assess what kind of testbed could be acceptable by the
majority (all if possible) of this and other matchmaking-related
communities. Hopefully, we should have a TREC-like effort, which would be
hard work, but surely worth of it.

As a final remark, I believe that in the following papers, ideas aimed at
facing some of the issues raised in this thread could be found:

1. A Non-Monotonic Approach to Semantic Matchmaking and Request Refinement
in E-Marketplaces [http://sisinflab.poliba.it/publications/2006/DDD06/]
2. A semantic-based fully visual application for matchmaking and query
refinement in B2C e-marketplaces
[http://sisinflab.poliba.it/publications/2006/DCDDRR06/] 

It is not a "Cicero pro domo sua" operation :) yet 
- in the first paper we propose how to use nonmonotonic reasoning services
as the basis to provide match explanations and to define ranking functions
- the second paper presents a prototype
(http://sisinflab.poliba.it/marketplace/) exploiting ideas presented in the
first paper and helping the user during query formulation and refinement.

I hope it is of some help for further useful discussion.

-- Tommaso

> -----Messaggio originale-----
> Da: public-sws-ig-request@w3.org 
> [mailto:public-sws-ig-request@w3.org] Per conto di Abraham Bernstein
> Inviato: venerdì 25 agosto 2006 16.00
> A: Matthias Klusch
> Cc: Terry Payne; Tommaso Di Noia; dellava@cefriel.it; 
> public-sws-ig@w3.org
> Oggetto: Re: R: sws matchmaker contest
> 
> 
> Dear all
> 
> Tomaso's and Terry's comments have prompted me to write down 
> some of my own thoughts.
> 
> I believe that there is a lot of benefit to look at 
> matchmaking in its greater context. As Terry points out there 
> are many research questions buried in how to best support 
> users in (i) query formulation, (ii) result set understanding 
> or re-ordering, and (iii) query refinement. 
> This is an area that is dear to my heart and I have spend 
> quite a bit of my research time on it over the passed two years.
> The same is true in the about the second field that Terry 
> points out, where no human is in the loop.
> 
> Nonetheless, I strongly believe that the development of a 
> benchmark should include tasks that disentangle the overall 
> loop (with the user or client program): in other words "pure" 
> retrieval/matchmaking  tasks. The reason is that I believe 
> that there are many different research tasks that should be 
> evaluated separately AS WELL AS in an integrated way.
> A quick brainstorm of tasks includes at least (I am probably 
> forgetting quite a few here):
> 
> 1) Given a set of queries and a collection of services,
>        a) how I can I find/rank the best matching ones?
>        b) how fast can I find a plausible one?
>        c) how do these matching procedures scale?
>        d) what is a suitable query language
>        e) what is the semantics of s partial match?
> 
> 2) Given a user and her/his need,
>       a) how can i help her/him put together a suitable query?
>       b) how can i help her/him understand the returned 
> answer set (and possibly its ranking)
>           and the trade-offs between the elements?
>      c) how can i help her/him improve/refine the query?
>      d) what is a suitable query language/formalism/...?
>      e)   ...
> 
> 3) <the same thing for agents/programs>....
> 
> 4) <tasks containing combinations of problems in 1, 2, and 3...>
> 
> Each of these tasks is worthwhile and should have some type 
> of test-collection.... eventually. But the question is where 
> to start. I would start with the first task, since it is the 
> one we all have the best understanding of (at least I believe so).
> We should then proceed with 2 and see what the multi-agent 
> community has done about 3...
> 
> So I am in great agreement with Terry and Tomaso, but just 
> wanted to point out that it may help to start with little 
> steps while keeping the overall goal in mind :-)
> 
> Cheers
> 
> Avi
> 
> 
> Matthias Klusch wrote:
> >
> >
> > dear all,
> >
> > thanks for the very useful feedback and hints to ongoing matchmaker 
> > development work so far!!
> >
> > one particular consequence of some of terry's notes, with which i 
> > agree, in essence, would be to build up a large sws retrieval test 
> > collection including domain(-independent) sws and user queries with 
> > subjectively defined relevance sets.
> > "Subjectively" implies to predominantly involve the 
> potential business 
> > domain service users in the iterative development process.
> > the hope is that such largely user driven design of a test 
> collection, 
> > in return, strongly influences the development of "pragmatically 
> > usable" sws matchmakers with "reasonably" good recall/precision 
> > performance on such a collection.
> >
> > i admit that this process does not, however, also 
> automatically lead 
> > to any solution of the problems related to "How can the user 
> > understand *why* the matchmaker returns what services?"
> > as Terry also noted in his last email.
> >
> > but it might be worth to start with the "(user) requirement 
> analysis 
> > phase for sws matchmakers, brokers, search engines". this is what I 
> > would expect to be a joint complementary action at the matchmaker 
> > contest meeting, basically triggering (or prepare to 
> trigger) the same 
> > kind of iterative user-research feedback driven development 
> process as 
> > happened with TREC a few decades ago.
> >
> > cordial regards, matthias
> >
> > __________________________________________________
> > Dr. Matthias Klusch
> > German Research Center for Artificial Intelligence 
> Stuhlsatzenhausweg 
> > 3
> > 66123 Saarbruecken, Germany
> > Phone: +49-681-302-5297, Fax: +49-681-302-2235 
> > http://www.dfki.de/~klusch/, klusch@dfki.de 
> > __________________________________________________
> >
> 
> --
> -----------------------------------------------------------------
> |  Professor Abraham Bernstein, PhD
> |  University of Zürich, Department of Informatics
> |  phone: +41 1 635 4579
> |  eMail: bernstein@ifi.unizh.ch
> |  web: www.ifi.unizh.ch/~bernstein
> |  mail: Binzmühlestrasse 14, CH-8050 Zürich, Switzerland
> 
>

Received on Friday, 25 August 2006 15:16:03 UTC