RE: FW: [Action-126] David to come up with a proposal for mtConfidence

I hope to get more feedback on the broader spec over the next week from other key MS stakeholders, but saw an opportunity to align us on this particular topic as quickly as possible with the MS Translator folks.



Jan

________________________________
From: Dr. David Filip [David.Filip@ul.ie]
Sent: Thursday, August 02, 2012 5:11 AM
To: Jan Nelson
Cc: public-multilingualweb-lt@w3.org
Subject: Re: FW: [Action-126] David to come up with a proposal for mtConfidence

Jan, thanks for checking with Chris and I am glad that you support the mtConfidence part.

Regarding the mt*Metrics parts.. I see what Chris means but I need to clarify and qualify..

First I think the comment is not valid for mtHumanMetrics part. What I mean are the simple score based human usability assessments that are increasingly used, but unfortunately the scales are not being standardized. AFAIK 4-5 value scales are being used, I would tend  to promote 4 values, as best practice, with  0 [publishable without changes] being the best, 3 [complete retranslate needed] the worst.

Regarding the mtAutomatedMetrics part, I agree that you need the pointers mentioned by Chris to PERFORM these. But that was NOT the goal of this part of the proposal. I simply assumed  that the MT producer has the metrics service set up (internally or as a third party service) and is able to provide it in runtime. I should have said so... The metrics-producing-service obviously must have all the mentioned pointers to perform the metrics, but IMHO this is not worth passing down..

Finally, I agree that the mtConfidence part seems most stable and best candidate for first wave standardization. If the other parts do not have support I am happy to drop them, I just wanted to chart the area in a general way..

Cheers
dF

Dr. David Filip
=======================
LRC | CNGL | LT-Web | CSIS
University of Limerick, Ireland
telephone: +353-6120-2781[X]
cellphone: +353-86-0222-158[X]
facsimile: +353-6120-2734[X]
mailto: david.filip@ul.ie<mailto:david.filip@ul.ie>



On Thu, Aug 2, 2012 at 7:36 AM, Jan Nelson <Jan.Nelson@microsoft.com<mailto:Jan.Nelson@microsoft.com>> wrote:

Feedback from Chris Wendt from our Microsoft Translator team on the section below:


-mtQuality
--mtConfidence
---mtProducer [string identifying producer Bing, DCU-Matrex etc.]
----mtEngine [string identifying the engine on one of the above platforms, can be potentially quite structured, pair domain etc.]
-----mtConfidenceScore [0-100% or interval 0-1]

All of the above makes total sense. I’d vote for it as proposed. Must be at sentence level, not below.

The following does not make sense without significant enhancements:

--mtAutomatedMetrics
---mtScoreType [METEOR, TER, BLEU, Levensthein distance etc.]
----mtAutomatedMetricsScore [0-100% or interval 0-1]
--mtHumanMetrics
---mtHumanMetricsScale [{4,3,2,1,0},{0,1,2,3,4}.{3,2,1,0} etc.]
----mtHumanMetricsValue [one of the above values depending on scale]


All of the scoring needs a pointer to one or more reference translations AND pointer to the source document. In addition, there may not be the exact same number of reference per element. You’d need to either add syntax for specifying the source and reference, or embed it all in the document. That’ll be messy.
These numbers are relevant only for MT system evaluations, which I consider a niche, and I would not try to standardize here in ITS.

Thanks,

Jan

________________________________
From: Felix Sasaki [fsasaki@w3.org<mailto:fsasaki@w3.org>]
Sent: Wednesday, August 01, 2012 12:50 AM
To: Dr. David Filip
Cc: public-multilingualweb-lt@w3.org<mailto:public-multilingualweb-lt@w3.org>

Subject: Re: [Action-126] David to come up with a proposal for mtConfidence
Hi David, all,
2012/7/31 Dr. David Filip <David.Filip@ul.ie<mailto:David.Filip@ul.ie>>
HI all, I was trying to engage a PhD student here at LRC to produce a proposal for this data category but I failed.

Nevertheless, here is my thinking on the category that maybe someone else (Declan?) could take it to the call for consensus stage.

co-chair hat on: If there is no strong support for this, I would propose to put this on hold until we have finished all other data categories. As you wrote in your agenda,

http://lists.w3.org/Archives/Public/public-multilingualweb-lt/2012Jul/0311.html

we have various data categories proposals on the table that are not finished: special requirements, named entity, quality, ... I will send a proposal for the time until last call later today, which will show that we need to finish these and the various "ed. notes" in
http://www.w3.org/International/multilingualweb/lt/drafts/its20/its20.html
I think we need the time and your input to work on these.

I very much hope for your understanding - let's discuss this also during the call on Thursday,

Felix


I believe that mtConfidence is being produced in some form or other by all major current MT systems. as discussed in Dublin, the issue is that these confidence scores are not really comparable between engines, I mean not only between Ging and Google, or Matrex, but even not between different pair engines or even specific domain trained engines based on the same general technology.

Nevertheless there are prospects for standardizing based on cognitive effort on post-editing etc. Even knowing that the usability of confidence scores is limited, there are valid production-consumption scenarios in the content lifecycle.
If a client/service provider/translator/reviewer do repeatedly work with the same engine, they will find even the engines self evaluation useful.

Further to this, there is potential of connecting this with automated and human MT evaluation scores, so I'd propose to generalize as mtQuality [mening raw MT quality, NOT talking about levels of PE] that would subume mtConfidence etc. as seen below

My proposal of the data model based on the above

-mtQuality
--mtConfidence
---mtProducer [string identifying producer Bing, DCU-Matrex etc.]
----mtEngine [string identifying the engine on one of the above platforms, can be potentially quite structured, pair domain etc.]
-----mtConfidenceScore [0-100% or interval 0-1]
--mtAutomatedMetrics
---mtScoreType [METEOR, TER, BLEU, Levensthein distance etc.]
----mtAutomatedMetricsScore [0-100% or interval 0-1]
--mtHumanMetrics
---mtHumanMetricsScale [{4,3,2,1,0},{0,1,2,3,4}.{3,2,1,0} etc.]
----mtHumanMetricsValue [one of the above values depending on scale]

mtQuality is an optional attribute of a machine text segment (as in Unicode or localization segmentations). I do not think this is useful on higher or lower levels.

mtQuality must be specified as mtConfidence XOR mtAutomatedMetrics XOR mtHumanMetrics

Then comes the compulsory specification the actual value (eventaully preceded by value change if more options exist)..

Cheers
dF


Dr. David Filip
=======================
LRC | CNGL | LT-Web | CSIS
University of Limerick, Ireland
telephone: +353-6120-2781<tel:%2B353-6120-2781>
cellphone: +353-86-0222-158<tel:%2B353-86-0222-158>
facsimile: +353-6120-2734<tel:%2B353-6120-2734>
mailto: david.filip@ul.ie<mailto:david.filip@ul.ie>




--
Felix Sasaki
DFKI / W3C Fellow




--
Felix Sasaki
DFKI / W3C Fellow