Re: MT Confidence definition

I'd agree with Yves here. There are systems that provide multiple candidate translations along with confidence estimates. Those estimates pertain to the target(s), not to the source. Tying them then to the source doesn't really make much sense to me since it is a report of how confident the system is that it has got it right. 

But as Yves notes, it's not even limited to self reporting. In the QTLaunchPad project we are working with quality estimation systems that provide a score after the fact for a system external to the tool. It is an independent assessment, so to speak. And this could be put in MT confidence (or in LQI, but I think MT confidence makes more sense). 

Best,

Arle

--
Arle Lommel
Berlin, Germany
Skype: arle_lommel
Phone (US): +1 707 709 8650

Sent from a mobile device. Please excuse any typos.

On Jul 17, 2013, at 12:35, Yves Savourel <ysavourel@enlaso.com> wrote:

> Hi Dave,
> 
> In the case of QuEst, for the scenario I have in mind, one would for example perform the MT part with MS Hub, then pass that
> information to QuEst and get back a score that indicate a level of confidence for that translation candidate. So that's a step after
> Mt and before any human looks at it.
> 
> I may be wrong, but "MT Confidence" seems to be a good place to put that information.
> 
> Even if QuEst is a wrong example. Having MT Confidence restricted to *self-reported* value seems very limiting. But maybe I'm mis
> interpreting the initial aim of the data category.
> 
> Cheers,
> -ys
> 
> -----Original Message-----
> From: Dave Lewis [mailto:dave.lewis@cs.tcd.ie] 
> Sent: Wednesday, July 17, 2013 12:25 PM
> To: public-multilingualweb-lt@w3.org
> Subject: Re: MT Confidence definition
> 
> Hi Yves,
> I don't necessarily agree with this based on the example you give in relation to quality estimation in Quest.
> 
> Is not the goal of quality estimation to predict the quality of a translation of a given source string for a given MT engine
> training corpora and training regime _prior_ to actually performing the translation?
> 
> In which case it would be an annotation of a translation but of a _source_ with reference to an existing or planned MT engine (which
> you rightly say in response to Sergey can be resolved via the annotatorsRef).
> 
> So while the basic data structure of mtConfidence would work for, the use case, name and wording don't i think match the use of MT
> QE.
> 
> Declan, Ankit could you comment - I'm not really an expert here, and not up to speed on the different applications of MT QE.
> 
> cheers,
> Dave
> 
> 
> On 17/07/2013 08:29, Yves Savourel wrote:
>> Hi all,
>> 
>> I've noticed a minor text issue in the specification:
>> 
>> For the MT Confidence data category we say:
>> 
>> "The MT Confidence data category is used to communicate the 
>> self-reported confidence score from a machine translation engine of the accuracy of a translation it has provided."
>> 
>> This is very limiting.
>> 
>> I think it should say:
>> 
>> "The MT Confidence data category is used to communicate the confidence 
>> score of the accuracy of a translation provided by a machine translation."
>> 
>> (and later: "the self-reported confidence score" should be "the reported confidence score").
>> 
>> There could be cases where the confidence score is provided by another 
>> system than the one that provided the MT candidate. The QuEst project 
>> is an example of this 
>> http://staffwww.dcs.shef.ac.uk/people/L.Specia/projects/quest.html)
>> 
>> Cheers,
>> -ys
> 
> 
>