Re: sws matchmaker contest

Hi Matthias,

I have realized that quite some effort was invested to create the
OWLS-TC, however as far as I am able to understand the descriptions, I
find the R/P evaluation not very expressive. This opinion is mainly
based up on the way the relevance sets are defined and the modeling that
has been chosen.

Let me try to explain my concerns using the first request/relevance set
defined. The goal is defined in the TC documentation as:
> The agent client is a lazy person who does request the price of a car, but didn't tell us its exact needs. Maybe the client is interested in a cheep/expensive/fast car, but it does not want to rent a car.
If I understand the TC correctly this goal is formally expressed in the
OWL-S file "car_price_service.owls". Within this file there seem to be 2
relevant parts:
A text description:
 <profile:textDescription xml:lang="en">
   This service returns price of a car.
  </profile:textDescription>
And the annotation of the input and outputs of the service, which
references to concepts in an accompanying ontology:
  <process:AtomicProcess rdf:ID="CAR_PRICE_PROCESS">
    <process:hasInput  rdf:resource="#_CAR"/>
   <process:hasOutput rdf:resource="#_PRICE"/>
  </process:AtomicProcess

If I now look into the relevance set, there is a file named
"car_report_service.owls". The relevant parts of it are:
  <profile:textDescription xml:lang="en">
    This service supply a critical report of analysis of a car,
    including its price also.
  </profile:textDescription>
[...]
  <profile:hasInput  rdf:resource="#_CAR"/>
  <profile:hasOutput rdf:resource="#_REPORT"/>

From the textual description I can completely understand why this
service is in the relevance set, however I have the following issues:
- You choose to capture the semantics of the service by characterizing
  its inputs and outputs as OWL concepts. While I agree that this is a
  viable way, there are also other options.
- You have invested some time in modeling the ontologies however, why
  did you decide to keep them incomplete? E.g. it could be easy modeled
  that a price is a part of a report.
- why are not all aspects of the request formally encoded, e.g. "... but
  it does not want to rent a car"
- The semantic degree of match in the relevance set is "failed". Since
  OWL uses an open world semantics and the ontologies do not state that
  a report and a price are disjoint it should at least be an
  intersection match?

> I disagree with that a test set would be automtically biased toward some
> approach and partly provie a solution. 
At the current state of affairs, I still believe every test set will do
so. Please correct me if I am wrong, but from above I have the
impression that the current OWL-S TC does describe information complete
in textual form but only incomplete in its semantic annotations. From
this I naively tend to believe that it is biased towards hybrid
approaches that take both text and explicit semantic annotation into
account.

> I would be very happy if you could provide us with the set
> of domain-specific OWL-S services with their WSDL grounding

As yet we do not have any submission that uses OWL-S. However all
submission are publicly available. You might for example want to take a
solution for the shipment discovery that has been done using F-Logic:
http://sws-challenge.org/2006/submission/polimi-cefriel-submission/discovery-scenario/

best
  Holger




Matthias Klusch wrote:
> Dear all,
> 
> I reckognise the rather opposite approach taken by the SWS challenge
> compared to that of the S3 contest of SWS matchmakers I am planning.
> Both have their value from different perspectives and goals in mind.
> 
> Some minor remarks to the response from the SWS challenge PC:
> 
>> Research is required how to actually make good representations: a test
>> set given in some formal semantics specification language will be
>> biased toward some specific approach (and be partly provide already a
>> solution to the overall challenge). 
> 
> I disagree with that a test set would be automtically biased toward some
> approach and partly provie a solution. In fact, it refers to the
> traditional challenge of creating a good test collection that can be
> used not only for a few selected use case scenarios like in your SWS
> challenge but many different domains like it has been successfully
> achieved with the TREC test collection in the text IR domain supported
> also by some major industrial stakeholders.
> 
> However, the opposite approach of the SWS challenge to start with few
> conventional WSDL services and let the participants actually try to
> generate and use their own appropriate *semantic* web services to
> find and improve the quality of the solution can, as a grass-root
> approach starting from WSDL contribute to the generation of a test
> collection for the S3 collection.
> 
>> Because of these differences in intent, we did not make a concrete
>> collaboration with Mathias' matchmaking workshop. However we believe
>> the community should not split. There should certainly be
>> synergy between these efforts.  We offer to use of our
>> open infrastructure for the matchmaking contest, in any way
>> that is useful.
> 
> Due to project related deadlines, I am planning with some
> colleagues in the EU funded CASCOM project to start a preliminary S3
> contest with four open source and different OWL-S matchmakers at DFKI
> hopefully next week. R/P evaluation will be done first by use of the
> OWLS-TC 2.1. We will of course keep you update on this and report on the
> results. However, these tests are to be considered preliminary only in
> that the main task of a real S3 contest has to be accomplished still,
> that is a large scale test collection for both OWL-S and WSML services,
> and even SA-WSDL, if possible.
> 
> As to the offer of the SWS challenge PC to help in this matter,
> I would be very happy if you could provide us with the set
> of domain-specific OWL-S services with their WSDL grounding
> (relevance set and service request) towards an extension of
> the open source OWLS-TC 2.1.
> 
> As to a meeting at ISWC, it seems to be hard if not impossible to
> synchronize with all interested people having conflicting
> travel arrangements and schedules for ISWC. Therefore I would
> like to suggest that anybody who is interested in the extension of
> OWLS-TC2 and the S3 contest should informally meet on Nov 7 right after
> Tom Gruber's invited talk in the coffee break, then we can
> quickly check how we can proceed from there (maybe during lunch, in the
> evening ?), and also to continue our discussion after the ISWC in the
> Wiki, Holger did set up for us.
> 
> best regards, matthias
> 
>>
>> regards
>>   SWS Challenge PC
>>   Charles, Michal, Holger
>>
> __________________________________________________
> Dr. Matthias Klusch
> German Research Center for Artificial Intelligence
> Stuhlsatzenhausweg 3
> 66123 Saarbruecken, Germany
> Phone: +49-681-302-5297, Fax: +49-681-302-2235
> http://www.dfki.de/~klusch/, klusch@dfki.de
> __________________________________________________
> 
> 

-- 
Holger Lausen

Digital Enterprise Research Institute (DERI)
http://www.deri.org/

Tel:   +43 512 5076464
Mail: holger.lausen@deri.org
WWW: http://holgerlausen.net