Re: sws matchmaker contest

Hi Holger,

measuring the traditional r/p performance is one way, and
has its well known deficiencies and strengths.

Holger Lausen schrieb:

> Hi Matthias,
> 
> I have realized that quite some effort was invested to create the
> OWLS-TC, however as far as I am able to understand the descriptions, I
> find the R/P evaluation not very expressive. This opinion is mainly
> based up on the way the relevance sets are defined and the modeling that
> has been chosen.
well, the point in here is that we had to start from the very scratch.
we are pretty much aware of the limits of the test collection as is, and 
this is exactly the motivation to improve on it by asking the community 
and others to help; otherwise the justified critic is a cyclic one
(it's bad therfore i do not join, therefore it remains bad ad infinitum:
bty, exactly the same experience has been made by the IR community
at the very early stage of TREC development).

> From the textual description I can completely understand why this
> service is in the relevance set, however I have the following issues:
> - You choose to capture the semantics of the service by characterizing
>   its inputs and outputs as OWL concepts. While I agree that this is a
>   viable way, there are also other options.
that's true, but, sorry, actually leads us to nowhere, since OWL-S bases
on OWL. of course you can use a different SDL such as WSML,
but OWLS-TC is a test collection for OWL-S matchmakers.
we are also trying to build one for WSMO-MX, as hard as it was for
OWLS-MX.... maybe eventually this time someone will join us in this
effort (sic!) the same problem as ever.

> - why are not all aspects of the request formally encoded, e.g. "... but
>   it does not want to rent a car"
in part because the collection was quickly developed (see also below)
and in part because it is not possible to have conditioned relations
between I/O parameters such as in WSML. but this is known for some time.

> - The semantic degree of match in the relevance set is "failed". Since
>   OWL uses an open world semantics and the ontologies do not state that
>   a report and a price are disjoint it should at least be an
>   intersection match?
 > - You have invested some time in modeling the ontologies however, why
 >   did you decide to keep them incomplete? E.g. it could be easy
 >  modeled that a price is a part of a report.
you are absolutely right, but i beg for your kind understanding that
this collection was developed initially very fast to get some
preliminary test results of our matchamker - there was nothing at
all, not even the smallest test collection for owl-s available.
as aa conxequence, too much work causes some failures; we want to 
improve on the work, but need help as we only have limited human
resources.
anyway, thanks for the feedback, I would definitely appreciate more of
it from more people from the community to push the joint building of a
service retrieval test collection  :-)

> As yet we do not have any submission that uses OWL-S. However all
> submission are publicly available. You might for example want to take a
> solution for the shipment discovery that has been done using F-Logic:
> http://sws-challenge.org/2006/submission/polimi-cefriel-submission/discovery-scenario/

thanks, but unfortunately this does not help us here for
our purpose of the s3 contest with focus on owl-s matchmakers.
but it could be of help for preparing the test of our WSMO-MX
matchmaker (using ontobroker for the f-logic reasoning).
anyway, i see the performance based contest of matchmakers and
your use case based contest as perfectly complementary even from
different views.

cordial regards, matthias

__________________________________________________
Dr. Matthias Klusch
German Research Center for Artificial Intelligence
Stuhlsatzenhausweg 3
66123 Saarbruecken, Germany
Phone: +49-681-302-5297, Fax: +49-681-302-2235
http://www.dfki.de/~klusch/, klusch@dfki.de
__________________________________________________

Received on Friday, 27 October 2006 15:29:20 UTC