Re: [Action-126] David to come up with a proposal for mtConfidence

Thanks Phil, while MT quality scores are obviously quality related, I do
not think there is much connection with quality assurance in the sense of
error catching and classification.

mtQuality/mtConfidence is intended as merely providing a (relative) score
per segment that is hoped to be eventually correlated in practice with
usefulness of the segment. If anyone sees a potential in subsuming this
into the quality model that is being worked on. I would not be against. I
just think that this is different in an important way and most probably
there would not be harm in keeping them separate. In the error based
quality model there should for instance be post-editting errors. But the
category as I proposed it is for raw MT only and should be archived or
deleted, once the raw MT has been modified or replaced..

Cheers
dF

Dr. David Filip
=======================
LRC | CNGL | LT-Web | CSIS
University of Limerick, Ireland
telephone: +353-6120-2781
*cellphone: +353-86-0222-158*
facsimile: +353-6120-2734
mailto: david.filip@ul.ie



On Tue, Jul 31, 2012 at 7:16 PM, Phil Ritchie <philr@vistatec.ie> wrote:

> David
>
> Maybe I'm off beam here but your narrative seems to express many of the
> challenges we've had defining the Quality Category - i.e. classification
> across distinct systems that produce the classifications. Whilst I think
> the data categories should be separate perhaps aligning with the philosophy
> of our attributes would solve or at least bring consistency in approach.
>
> Phil
>
>
>
> On 31 Jul 2012, at 18:48, "Dr. David Filip" <David.Filip@ul.ie> wrote:
>
> HI all, I was trying to engage a PhD student here at LRC to produce a
> proposal for this data category but I failed.
>
> Nevertheless, here is my thinking on the category that maybe someone else
> (Declan?) could take it to the call for consensus stage.
>
> I believe that mtConfidence is being produced in some form or other by all
> major current MT systems. as discussed in Dublin, the issue is that these
> confidence scores are not really comparable between engines, I mean not
> only between Ging and Google, or Matrex, but even not between different
> pair engines or even specific domain trained engines based on the same
> general technology.
>
> Nevertheless there are prospects for standardizing based on cognitive
> effort on post-editing etc. Even knowing that the usability of confidence
> scores is limited, there are valid production-consumption scenarios in the
> content lifecycle.
> If a client/service provider/translator/reviewer do repeatedly work with
> the same engine, they will find even the engines self evaluation useful.
>
> Further to this, there is potential of connecting this with automated and
> human MT evaluation scores, so I'd propose to generalize as mtQuality
> [mening raw MT quality, NOT talking about levels of PE] that would subume
> mtConfidence etc. as seen below
>
> My proposal of the data model based on the above
>
> -mtQuality
> --mtConfidence
> ---mtProducer [string identifying producer Bing, DCU-Matrex etc.]
> ----mtEngine [string identifying the engine on one of the above platforms,
> can be potentially quite structured, pair domain etc.]
> -----mtConfidenceScore [0-100% or interval 0-1]
> --mtAutomatedMetrics
> ---mtScoreType [METEOR, TER, BLEU, Levensthein distance etc.]
> ----mtAutomatedMetricsScore [0-100% or interval 0-1]
> --mtHumanMetrics
> ---mtHumanMetricsScale [{4,3,2,1,0},{0,1,2,3,4}.{3,2,1,0} etc.]
> ----mtHumanMetricsValue [one of the above values depending on scale]
>
> mtQuality is an optional attribute of a machine text segment (as in
> Unicode or localization segmentations). I do not think this is useful on
> higher or lower levels.
>
> mtQuality must be specified as mtConfidence XOR mtAutomatedMetrics
> XOR mtHumanMetrics
>
> Then comes the compulsory specification the actual value (eventaully
> preceded by value change if more options exist)..
>
> Cheers
> dF
>
>
> Dr. David Filip
> =======================
> LRC | CNGL | LT-Web | CSIS
> University of Limerick, Ireland
> telephone: +353-6120-2781
> *cellphone: +353-86-0222-158*
> facsimile: +353-6120-2734
> mailto: david.filip@ul.ie
>
>
> ************************************************************
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the sender immediately by e-mail.
>
> www.vistatec.com
> ************************************************************
>