Re: [Action-126] David to come up with a proposal for mtConfidence

David

Maybe I'm off beam here but your narrative seems to express many of the challenges we've had defining the Quality Category - i.e. classification across distinct systems that produce the classifications. Whilst I think the data categories should be separate perhaps aligning with the philosophy of our attributes would solve or at least bring consistency in approach.

Phil



On 31 Jul 2012, at 18:48, "Dr. David Filip" <David.Filip@ul.ie> wrote:

> HI all, I was trying to engage a PhD student here at LRC to produce a proposal for this data category but I failed.
> 
> Nevertheless, here is my thinking on the category that maybe someone else (Declan?) could take it to the call for consensus stage.
> 
> I believe that mtConfidence is being produced in some form or other by all major current MT systems. as discussed in Dublin, the issue is that these confidence scores are not really comparable between engines, I mean not only between Ging and Google, or Matrex, but even not between different pair engines or even specific domain trained engines based on the same general technology.
> 
> Nevertheless there are prospects for standardizing based on cognitive effort on post-editing etc. Even knowing that the usability of confidence scores is limited, there are valid production-consumption scenarios in the content lifecycle.
> If a client/service provider/translator/reviewer do repeatedly work with the same engine, they will find even the engines self evaluation useful.
> 
> Further to this, there is potential of connecting this with automated and human MT evaluation scores, so I'd propose to generalize as mtQuality [mening raw MT quality, NOT talking about levels of PE] that would subume mtConfidence etc. as seen below
> 
> My proposal of the data model based on the above
> 
> -mtQuality
> --mtConfidence
> ---mtProducer [string identifying producer Bing, DCU-Matrex etc.]
> ----mtEngine [string identifying the engine on one of the above platforms, can be potentially quite structured, pair domain etc.]
> -----mtConfidenceScore [0-100% or interval 0-1]
> --mtAutomatedMetrics
> ---mtScoreType [METEOR, TER, BLEU, Levensthein distance etc.]
> ----mtAutomatedMetricsScore [0-100% or interval 0-1]
> --mtHumanMetrics
> ---mtHumanMetricsScale [{4,3,2,1,0},{0,1,2,3,4}.{3,2,1,0} etc.]
> ----mtHumanMetricsValue [one of the above values depending on scale]
> 
> mtQuality is an optional attribute of a machine text segment (as in Unicode or localization segmentations). I do not think this is useful on higher or lower levels.
> 
> mtQuality must be specified as mtConfidence XOR mtAutomatedMetrics XOR mtHumanMetrics
> 
> Then comes the compulsory specification the actual value (eventaully preceded by value change if more options exist)..
> 
> Cheers
> dF
> 
> 
> Dr. David Filip
> =======================
> LRC | CNGL | LT-Web | CSIS
> University of Limerick, Ireland
> telephone: +353-6120-2781
> cellphone: +353-86-0222-158
> facsimile: +353-6120-2734
> mailto: david.filip@ul.ie
> 

************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender immediately by e-mail.

www.vistatec.com
************************************************************