Re: sws matchmaker contest

dear colleagues,

as promised to keep you updated on the process of preparing 
a first s3 contest and test collection for sws:

there has been a brief informal 30minutes meeting on this topic 
at iswc 2006 conference in athens, usa. 
the attendees were 
terry payne (u southampton, uk), 
alain leger (telecom france, f), 
avi bernstein (u zürich, ch),
massimo paolucci (docomo research labs europe, d),
david martin (sri)
and myself.
(pls take my apologies if i forgot someone to list, 
it was very busy at this time with people joining/leaving the
 discussion)

after intense discussion of related issues, we agreed to propose
following a kind of a grass root approach for building up
a sws retrieval test collection in the style of trec in the ir domain.
even if this may take a long time to eventually succeed, as with trec,
for several discussed reasons we strongly believe that it is worth it 
to start right now to enable an evolutionary development 
and gradual improvement of s3 tools like the matchmakers 
already available. if the whole approach fails, we know at 
least why in much more concrete terms than pure speculations.

(1) the idea was to continously collect sws - sa-wsdl, owl-s or wsml -
we all
come across during our work, and send it to a central collector site for
   editing the respective *raw collections*. 

i volunteer for this task but would be perfectly happy if someone else
   is keen to do it.

(2)domain oriented and domain independent reference queries can be 
  acquired by different means, by the research community and
  by industry partners with domain experts in our projects
  that are interested in deploying sws, or that are already using 
  sws in house, in projects, etc.
  for this purpose, potential users shall be proactively 
  contacted, as well as invited to activily participate in the
  process also during the meetings.

(3) the raw sws collections then should be jointly structured 
  - domain, query relevance sets assignments - on a regular basis 
  at respective workshops or meetings at conferences
  to minimize costs.

(4) as with trec, the gradually extended collections would then be used
  to test the quality of s3 tools at respective s3 contests.
  
we would to start the collection phase now, 
hence would like to very cordially invite you to send us the sets of
services you have at hand in your projects - be it wsml, sa-wsdl, or
owl-s -
in a zipped format whenever but as soon as possible.

i will make the raw collection available on a web page regularly 
to minimize redundant submissions.  

pls note that these collections are still not the retrieval test
collections,
as the updated set of queries and relevance sets to be decided upon
jointly are still missing (see (2)).

we are currently preparing a s3 workshop proposal
to be submitted to eswc 2007, that means if successfull, we might have
a first official s3 meeting with initially designed collections and 
then first s3 contest using them at this workshop by mid 2007.

looking forward to your reply, and maqbe even first submissions of 
sws sets :-)

cordial regards, matthias

ps: concurrently, at dfki we are preparing a test environment
for comparatively testing the performance of different matchmakers for
owl-s
services available using owls-tc 2.1. (currently: pcem (adetti), owls-mx
(dfki), rowls (urjc), and owls-uddi (cmu). we would be very happy to
test others as
well! so, pls let us know!). if someone could do that also 
for wsml or sa-wsdl that would be great!

--------------------------------------------------
Dr Matthias Klusch
German Research Center for Artificial Intelligence
Stuhlsatzenhausweg 3, 66123 Saarbruecken, Germany
phone: +49-682-3025297
web: http://www.dfki.de/~klusch  
--------------------------------------------------

Received on Sunday, 19 November 2006 13:53:26 UTC