RDFTM: Evaluation criteria

(This email was posted on February 02 to the survey editors as part of
an email thread among the editors and myself. I'm reposting it here by
request of Steve Pepper.)

* Steve Pepper
| 
| I find that the Unibo paper from Extreme Markup contains too few
| details. 

I can only agree with that myself. I read it again when the RDFTM work
started up in order to evaluate it, but couldn't find sufficient
detail about how it actually worked to do that. In fact, unless I'm
mistaken, the survey description contains more information than the
Montréal paper.

| I am still not entirely happy with the five criteria. In particular
| I found "reversibility" difficult to handle when updating Stanford
| and Ogievetsky. On the one hand, any one-way translation is
| theoretically reversible, provided no information is lost (and
| information loss is covered by the criterion "completeness").

The key point here is probably that the survey seems to take the
position that RDF->TM and TM->RDF conversions are dependent on each
other. I strongly disagree with that, and think that a good X->Y
conversion will be able to convert any X to the appropriate Y
regardless of whether the origin of the X was human authoring or Y->X
conversion.

I think this leads to serious confusion, and here's my attempt to
clear that up (all the time assuming the independence of X->Y from
Y->X).

If a conversion is complete, the conversion will be reversible so long
as *some* complete conversion in the other direction exists.

If a conversion is reversible (given *some* complete conversion in the
other direction), the conversion will be complete.

In other words: reversibility and completeness are equivalent. That
means that they are the *same* criterion, expressed in two different
ways.

I think that for users it is more important to have completeness (loss
of information is unacceptable) than to have reversibility (they may
not want to go back to where they came from). 

Therefore, what I think we should do is to remove the reversibility
criterion, but to, in the description of completeness, use
reversibility as the test for completeness, since it is easier to
verify in an objective manner.

| On the other hand, if a proposal covers two-way translations, what
| we are perhaps really interested in is whether it provides for
| roundtripping.

I think we are *always* interested in whether it provides for
roundtripping (whether on its own or when supported by some other
proposal), because if it does not it isn't complete.

I also think that the criteria should be the same regardless of
whether the proposal is one-way or two-way, since the TF is looking
for proposals that meet our criteria, and the packaging of the
proposals is irrelevant for our purposes.

In fact, the cleanest way to approach this might be to break all the
proposals up into

  Proposer RDF->TM
  Proposer TM->RDF
 
proposals, and then evaluate each on its own. My rationale for this is
that what we are looking for is one RDF->TM conversion and one TM->RDF
conversion, and theoretically we could wind up with the RDF->TM
conversion from one proposal and the TM->RDF conversion from another.

If nothing else, I think doing this would simplify the evaluations and
make the situation as whole much clearer. It would also remove the
"Direction" criterion entirely, which I think is all to the good,
really.

| So perhaps the criterion should be roundtripping and it should only
| apply for approaches that cover both directions?

Well, let's imagine the following situation:

  proposal X is TM->RDF only, and objectively the best for this direction
  proposal Y is RDF->TM only, and objectively the best for this direction
  proposal Z is two-way, and objectively slightly less good than X & Y

With the current criteria (as well as with your new proposal) X and Z
would score low on direction/roundtripping, while Z would get full
marks. If they are close on the other criteria, except that Z scores
lower on, say, fidelity, Z might emerge as overall winner.

What should we do then? Choose X and Y, or Z? Clearly, the correct
answer is X and Y, but our criteria don't point us in that direction.

I guess my question is why we would create this situation at all. Why
not just remove the direction criterion entirely, separate the apples
from the oranges, then pick the best apple and the best orange?

-- 
Lars Marius Garshol, Ontopian         <URL: http://www.ontopia.net >
GSM: +47 98 21 55 50                  <URL: http://www.garshol.priv.no >

Received on Friday, 18 February 2005 08:29:26 UTC