RE: Call for consensus - Localization Quality Précis (related to [ISSUE-34])

You explained this very well Felix, thanks.

 

So I think there are several directions we can take:

 

a)  We can keep the score to be a range and not address at all the ‘voting’ systems.

b)  We can change the score value and meaning to work for ‘voting’ only and stop addressing range systems.

c)  We can add a new attribute that supports the voting systems

d)  We can change the score to be an undefined value and make the profile reference mandatory, and declare that the score has its meaning bounded to the given profile. Only tools knowing the profile intent will be able to truly use the score then (Des’ initial solution I think).

 

C Would probably be the best solution, that is if we can define a standard way to express voting values (and find enough implementations?)

 

D Would be similar to the mtConfidence proposal where we bound the value to a tool/engine: The values are comparable only when they are bounded to the same profile. General interoperability is limited to basically displaying the value, and the tools can truly interact with the value only if they know specifically what is the meaning of the values for that profile.

 

-ys

 

 

 

From: Felix Sasaki [mailto:fsasaki@w3.org] 
Sent: Friday, August 24, 2012 9:26 AM
To: Des Oates
Cc: Yves Savourel; public-multilingualweb-lt@w3.org
Subject: Re: Call for consensus - Localization Quality Précis (related to [ISSUE-34])

 

Hi Des,

 

I think the point is not whether you have a range +/- 100 or only +100, but the difference between a range and voting like user evaluation which you had described before.

 

With the range approach, if the value of the locQualityPrecisScore is 80, you can say: the system scored 80% of a maxium.

 

With the user voting you described with the comparison to youtube, the value 80 doesn't give you the information "80 is 80% of maxium 100". It just tells you: the total sum of user votings is 80.

 

If we say that a value like 80 can have both the voting and the range interpretation, applications which want to process locQualityPrecisScore probably won't know what to do. E.g. if in a workflow there is a step "everything below 80% quality needs a check", that makes only sense for the range interpretation, but not for the voting approach.

 

Best,

 

Felix

2012/8/24 Des Oates <doates@adobe.com>

Yves, yes in theory it is open ended but a range of +/- 100 would satisfy most use cases. (The vast majority of candidate translations vote counts I have seen to date on are in the 10s range or less, not the 100s).  If not then implementers could put in some form of logarithmic mapping with limits at +/- 100.  I don't think it is necessary to add another attribute specifically for this at this point. I think it is sufficient just to overload the locQualityPrecisScore as mentioned before, at least for now.

On saying that, a standalone voting attribute has merits, since it could capture other information along with the aggregate score such as  total '+' counts, total '-' counts which are also useful metrics.  However I think that is out of scope for ITS2.0.

Cheers

Des


-----Original Message-----
From: Yves Savourel [mailto:ysavourel@enlaso.com]

Sent: 24 August 2012 14:15
To: public-multilingualweb-lt@w3.org
Subject: RE: Call for consensus - Localization Quality Précis (related to [ISSUE-34])

Hi Des,

> This particular segment has 3 translation candidates that users can
> vote on. Users can click either the checkbox (+ve vote) or the ‘x’
> (-ve vote).  The aggregate of all users’ votes for each candidate
> translation represents the ‘quality score’ for that candidate.

So, just to see if I get this right: if, for a given translation, you have 5 'plus' and 12 'minus', your aggregated score will be -7?

If that's the case, your system seems to be indeed open-ended and cannot be truly mapped to a range.

Therefore I don't think having locQualityPrecisScore as value between -100 and 100 will be more useful than if it's a value between 0 and 100. In both case it wouldn't be interoperable with any of the range-based scores used by other systems.

Basically, I think a range-based value can only represent a rating, not a voting system.

Maybe we need another attribute altogether that would represent voting?

-yves









 

-- 
Felix Sasaki

DFKI / W3C Fellow