Draft of quality section

There is a lot to be done still, but I did want to circulate the current draft of the loc-quality section of the document (Yves, you'll notice I accepted that suggestion). I'm sure a lot of "scaffolding" still shows in this document, but it's getting closer to a working model, so I invite feedback on this text. The next step will be to implement suggestions and then add it to the actual spec.

Note to Yves: I have not yet implemented the suggestion for a resolution status aspect and I still need to work on the global markup. Felix and I also discussed the XLIFF 2.0-style markup you had suggested, and that needs more discussion. But at this point, do we think we more or less have the appropriate categories and information? Also, if you would check the mapping I did from CheckMate to the quality types list and let me know if it looks right to you, I would appreciate it.

Thanks,

Arle


6.x ITS 2.0 Localization Quality

The Localization Quality (loc-quality) data category expresses information related to localization quality assessment tasks. It is intended to provide guidance about about quality issues for human reviewers as well as quantitative information that may be used in a localization workflow to make appropriate workflow decisions.

Because of the relative complexity of localization quality assurance tasks, this data category consists of a number of subcategories, as described in the table presented below. These are designed to work in tandem with each other, but not all categories will be used in all situations.

Conformance NOTE: Tools implementing this data category MUST support loc-quality-profile AND either loc-quality-type OR loc-quality-comment. Only in the case of manual human annotation for quality issues may loc-quality-profile be omitted. In addition, if a tool produces internal codes, it is strongly recommended that they support loc-quality-code as well. The loc-quality-score and loc-quality-severity subcategories are optional and are used by tools that support these features.

6.x.1 Subcategory Descriptions

Subcategory name Description Permissible values Notes
loc-quality-profile Pointer to a description of the quality assessment model in use, with a description of the categories a tool-specific prefix, optionally followed a colon (:) and a URI where the description of the tools quality categories is available. The use of a URI is strongly recommended as it provides a way for human evaluators to learn more about the quality categories in use.
loc-quality-score The score value generated by a quality assessment process integer value from 0 to 100. Higher values equal better scores. Users would need to normalize internal scores to match this system upon generation and convert these scores to match their own internal system upon consumption.
loc-quality-type top-level quality type, as defined in the specification picklist with values drawn from the list of loc-quality-type values ITS 2.0-compliant tools that use these categories would need to map their internal values to these types
loc-quality-code At internal classification code for a quality issue as produced by the generating tool qname + text The prefix used in the qname MUST correspond to the tool-specific prefix declared in loc-quality-profile. In cases where other ITS loc-quality subcategories apply to the same document node, the prefix used in this value is assumed to apply to all other subcategories. Therefore, if this value does not apply to them, they MUST be applied to a separate element (e.g., a separate HTML <span> element). (NB: need an example of this.)
loc-quality-comment A human-readable description of the quality issue text Use of loc-quality-comment would be strongly recommended in any cases where the value of other is used forloc-quality-type.
loc-quality-severity A numerical value representing the severity of the issue, as defined by the model generating the metadata number from 0 to 1 with up to two decimal places, with higher values equaling greater severity It is up to tools to map the numerical values of this to their own system. We can provide some informative guidelines for how this is to be done based on internal severity systems.
Possible attributes (we are less certain on these)
loc-quality-stage A value to indicate the status of a particular issue in a review workflow Plain text The values here are particular to specific workflows and may not be interoperable with other workflows
loc-quality-threshold A value which defines apassing score for loc-quality-score integer value from 0 to 100. A value of loc-quality-score greater than or equal to the value of loc-quality-threshold is deemed to havepassed the quality assessment process.
It may make sense to leave this as part of the description referred to in loc-quality-profile, but having it here would allow processes to automate actions based on whether the file passes or not.
loc-quality-agent An identifier for the agent that produced the quality results ??? Perhaps a picklist withhuman and machine as values? Needs better definition
Localization Quality Subcategories, Descriptions, and Permissible Values
6.x.1.1 Values of loc-quality-type

The loc-quality-type subcategory is intended to provide a basic level of interoperability between differing localization quality assurance systems. It provides a list of 26 high-level quality issue types common in automatic and human localization quality assessment. Localization quality assessment tools can map their internal categories to these categories in order to exchange information about the kinds of issues they identify and take intelligent and appropriate action even if another tool does not know the specific issues identified by the generating tool

NOTE: Tools implementing the loc-quality data category that use the loc-quality-code subcategory SHOULD also use the loc-quality-code to provide this level of interoperability.

The values listed in the following table are allowed for loc-quality-type. Note that tools implementing loc-quality-type are not required to check for or flag issues for all or any of the types. However, if they implement loc-quality-type, the values they produce for the attribute MUST match one of the values provided in this table and MUST be semantically accurate. If a tool can map its internal values to these categories it MUST do so and may not use the value of other, which is reserved strictly for values that cannot be mapped to these values.

Allowable values for loc-quality-type
Value Description Examples Notes
terminology An incorrect term or a term from the wrong domain was used or terms are used inconsistently 
The localization had Pen Drive when corporate terminology specified thatUSB Stick was to be used; The localization inconsistently used Startand Begin.
Should not be confused with the ITS terminology data category.
mistranslation The content of the target mistranslates the content of the source 
The English source reads “An ape succeeded in grasping a banana lying outside its cage with the help of a stick” but the Italian translation reads “l’ape riuscì a prendere la banana posta tuori dall sua gabbia aiutandosi con un bastone” (“A bee succeeded…”)
Issues related to translation of specific terms related to the domain or task-specific language should be categorized as terminology issues
omission Necessary text has been omitted from the localization or source 
One or more segments found in the source that should have been translated are missing in the target
This category should not be used for missing whitespace or formatting codes, but instead should be reserved for linguistic content.
untranslated Content that should have been translated was left untranslated 
The source segment reads “The Professor said to Smith that he would hear from his lawyer” but the Hungarian localization reads “A professzor azt modta Smithnek, hogy he would hear from his lawyer.”
omission take precedence over untranslated. Omissions are distinct in that they address cases where text is not present, while untranslated address cases where text has been carried from the source untranslated.
addition The translated text contains inappropriate additions 
The translated text contains a note from the translator to himself to look up a term; the note should have been deleted but was not.
duplication Content has been duplicated improperly 
A section of the target text was inadvertently copied twice in a copy and paste operation.
inconsistency The text is inconsistent with itself (NB: not for use with terminology inconsistency) 
The text states that an event happened in 1912 in one location but in another states that it happened in 1812.
grammar The text contains a grammatical error (including errors of syntax and morphology) 
The text reads “The guidelines says that users should use a static grounding strap.”
legal The text is legally problematic (e.g., it is specific to the wrong legal system) 
The localized text is intended for use in Thailand but includes U.S. regulatory notices.
A text translated into German contains comparative advertising claims that are not allowed by German law
register The text is written in the wrong linguistic register of uses slang or other language variants inappropriate to the text 
A financia text translated into U.S. English refers to dollars as “bucks”.
locale-specific-content The localization contains content that does not apply to the locale for which it was prepared 
A text translated for the Japanese market contains call center numbers in Texas and refers to special offers available only in the U.S.
Legally inappropriate material should be classified as legal
locale-violation Text violates norms for the intended locale 
A text localized into German has dates in YYYY-MM-DD format instead of in DD.MM.YYYY
A translated text uses American-style foot and inch measurements instead of centimeters.
style The text contains stylistic errors 
Company style dictates that all individuals be referred to as Mr. or Ms. with a family name, but the text refers to “Jack Smith”.
characters The text contains characters that are garbled or incorrect or that are not used in the language in which the content appears 
the text should have a
but instead has a ¥ sign
A text translated into German omits the umlauts over ü, ö, and ä
A Japanese localization contains characters like మ and ఊ (from Telugu)
misspelling The text contains a misspelling 
A German text misspells the wordZustellung as Zustellüng
typographical The text has typographical errors such as omitted/incorrect punctuation, incorrect capitalization, etc. 
An English localization has the following sentence: The man whom, we saw, was in the Military and carried it’s insignias
formatting The text is formatted incorrectly 
Warnings in the target text are supposed to be set in italic face, but instead appear in bold face
Margins of the text are narrower than specified
inconsistent-entities The source and target text contain different named entities (dates, times, place names, individual names, etc.) 
The name Thaddeus Cahill appears in an English source but is rendered asTamaš Cahill in the Czech version
The date February 9, 2007 appears in the source but the translated text has “2. September 2007.”
numbers Numbers are inconsistent between source and target 
The source text states that an object is 120 cm long, but the target text says it is 129 cm. long.
Some tools may correct for differences in units of measurement to reduce false positives
markup There is an issue related to markup or a mismatch in markup between source and target 
The source segment has five markup tags but the target has only two
An opening tag in the localization is missing a closing tag
pattern-problem The text fails to match a pattern that defines allowable content (or matches one that defines non-allowable content) 
The quality checking tool disallows the regular expression pattern ['"”’][\.,]but the translated text contains A leading “expert”, a political hack, claimed otherwise.
whitespace There is a mismatch in whitespace between source and target content 
A source segment starts with six space characters but the corresponding target segment has two non-breaking spaces at the start.
internationalization There is an error related to the internationalization of content 
A line of programming code has embedded language-specific strings
A user interface element leaves no room for text expansion
A form allows only for U.S.-style postal addresses and expects five digit U.S. ZIP codes
There are many kinds of internationalization errors of various types. This category is therefore very heterogeneous in what it can refer to.
length There is a significant difference in source and target length 
The translation of a segment is five times as long as the source
What constitutes a “significant” difference in length is determined by the model referred to in theloc-quality-profile
uncategorized The issue has not been categorized 
A new version of a tool returns information on an issue that has not been previously checked and that is not yet classified
This category has to uses: (1) a tool can use it to pass through quality data from another tool in cases where the issues from the other tool are not classified (for example, a localization quality assurance tool interfaces with a third-party grammar checker); (2) a tool’s issues are not yet assigned to categories, and, until an updated assignment is made, they may be listed as uncategorized. In the latter case it is recommended that issues be assigned to appropriate categories as soon as possible since uncategorized does not foster interoperability.
other Any issue that cannot be assigned to any values listed above.  This category allows for the inclusion of any issues not included in the previously listed values. This value MUST not be used for any tool- or model-specific issues that can be mapped to the values listed above. In addition, this value is not synonymous with uncategorized in that uncategorized issues may be assigned to another precise value, while other issues cannot.

If a metric has an “miscellaneous” or “other” category, it should be mapped to this value even if the specific instance of the issue might be mapped to another category.
Example of localization-quality local markup in HTML5

The following example uses local HTML5 markup with CSS to highlight quality issues in browser rendition of the document. It uses a fictional DFKI tool in the markup and the markup should not be interpreted as referring to an actual quality assurance system.

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Telharmonium 1897</title>
    <meta name="its-localization-quality-profile"
        content="dfki:http://www.dfki.de" />
    <meta name="its-localization-quality-score" content="56" />
    <style type="text/css">
      [its-localization-quality-type]{
        border:1px solid green;
        margin:2px;
      }
      [its-localization-quality-type = untranslated] {
        background-color:red;
      }
      [its-localization-quality-type = whitespace] {
        background-color:yellow;
      }
      [its-localization-quality-type = inconsistent-entities]{
        background-color:#9DFFE1;
      }
      [its-localization-quality-type = spelling]{
        background-color:#FFE2F7;
      }
      [its-localization-quality-severity = "1.0"]{
        border: 6px solid red;
      }
    </style>

  </head>
  <body>
    <h1 id="h0001" its-localization-quality-type="untranslated"
      its-localization-quality-code="dfki:target_equals_source">
      Telharmonium (1897)</h1>
    <p id="p0001">
      <span class="segment" id="s0001">
        <span its-localization-quality-type="inconsistent-entities"
          its-localization-quality-code="dfki:named_entity_not_found"
          its-localization-quality-note="Should be Thomas Cahill. Why
            is Batman in the picture?"
          its-localization-quality-severity="1.0">Christian Bale</span>
        <span its-localization-quality-type="whitespace"
          its-localization-quality-code="dfki:extra_space_around_punctuation"
          its-localization-quality-severity="0.1">(1867 – 1934)</span>
        conceived of an instrument that could transmit its sound from a power
        plant for hundreds of miles to listeners over telegraph wiring.</span>
      <span class="segment" id="s0002">Beginning in 1889 the sound quality of
        regular telephone concerts was very poor on account of the buzzing
        generated by carbon-granule microphones. As a result Cahill decided to
        set a new standard in perfection of sound
        <span its-localization-quality-type="spelling"
          its-localization-quality-code="dfki:spelling_error"
          its-localization-quality-severity="0.5"
          its-localization-quality-note="should be 'quality'">qulaity</span>
        with his instrument, a standard that would not only satisfy listeners
        but that would overcome all the flaws of traditional instruments.</span>
    </p>
  </body>
</html>
And here is the rendered example that shows how the CSS is used to highlight issues:

Telharmonium (1897)
Christian Bale (1867 – 1934) conceived of an instrument that could transmit its sound from a power plant for hundreds of miles to listeners over telegraph wiring. Beginning in 1889 the sound quality of regular telephone concerts was very poor on account of the buzzing generated by carbon-granule microphones. As a result Cahill decided to set a new standard in perfection of sound qulaity with his instrument, a standard that would not only satisfy listeners but that would overcome all the flaws of traditional instruments.

Annex X: Mapping of Tool-Specific Quality Codes to loc-quality-type Values (Non-Normative)

This Annex is informative.

The following table provides mappings of native quality assurance issue codes for a number of common localization quality tools to loc-quality-type values. Tool developers are free to map their own issue codes to the loc-quality-type values and are encouraged to make their mappings publicly available. Tools that produce ITS 2.0 loc-quality markup should ensure that the output of their tools matches any publicly available mappings they may produce.

Note: These mappings are provides for example only. In the event of discrepancy between the mapping published by a developer and this annex, the statements from the developer take precedence over this annex.

loc-quality-typevalue Tool/Metric-Specific Values
CheckMate xliff:doc QA Distiller SAE J2450 LISA QA Model (UI) LISA QA Model (doc) - language only*
terminology 
TERMINOLOGY
terminology
Consistency
Tag-aware
ID-aware
Untranslatables
wrong term
Terminology
Glossary adherence
Abbreviations
Context
mistranslation   
Mistranslation
Accuracy
Semantics
Accuracy
omission 
MISSING_TARGETTU
MISSING_TARGETSET
EMPTY_TARGETSEG
EMTPY_SOURCESEG
omission
Empty translations
omission
Omissions
untranslated 
TARGET_SAME_AS_SOURCE
Forgotten translations
Skipped translations
Partial translations
Incomplete translation
addition 
EXTRA_TARGETSEG
Additions
duplication Not addressed in any of these metrics. It may be possible to treat this as a case of addition.
inconsistency  
inconsistency
Source
Target
Consistency
grammar    
syntactic error
word structure or agreement error
Grammar
legal Not addressed in any of these metrics. However, legal compliance checking is a big deal for regulated industries and forms a core part of their metrics.
register      
Register/tone
Language variants/slang
locale-specific-content      
Local suitability
locale-violation     
Country
Country standards
style     
Style
General style
Company standards
characters 
ALLOWED_CHARACTERS
Corrupt characters, source
Corrupt characters, target
Double/Single Size
Character formatting
misspelling    
misspelling
Spelling
typographical  
punctuation
Consecutive punctuation marks
End of segment punctuation
Non-matching pairs (brackets)
Leading bracket outside of TU
Different count(brackets)
Initial capitalization
Entire capitalization
Non-matching pairs (quotation marks)
Incorrect type (quotation marks)
Different count (quotation marks)
punctuation error
Punctuation marks
formatting     
TOC
Index
Layout
Typography
Graphics
Call Outs and Captions
Alignment
Sizing
Truncation/overlap
(Numerous)
inconsistent-entities  
date
time
numbers  
number
Number values
Incorrect type (measurements)
Check conversions (measurements)
markup 
MISSING_CODE
EXTRA_CODE
SUSPECT_CODE
tags
pattern-problem 
UNEXPECTED_PATTERN
SUSPECT_PATTERN
pattern
whitespace 
MISSING_LEADINGWS
MISSINGORDIFF_LEADINGWS
EXTRA_LEADINGWS
EXTRAORDIFF_LEADINGWS
MISSING_TRAILINGWS
MISSINGORDIFF_TRAILINGWS
EXTRA_TRAILINGWS
EXTRAORDIFF_TRAILINGWS
Consecutive spaces
Inconsistent leading and trailing spaces
Required/forbidden spaces
Different count (tabs)
Required/forbidden spaces
internationalization  
internationalization*
(*The examples for this code are broader than the type category here. Nevertheless, it makes sense to maintain this mapping.)

Number formatting
length 
TARGET_LENGTH
uncategorized 
LANGUAGETOOL_ERROR
other  
other
miscellaneous error
Hyper text functionality, jumps, popups
Localizable text
Dialogue functionality
Menu functionality
Hotkeys/accelerators
Jumps/links
(* The LISA QA Model documentation addresses numerous issues related to software formatting that are outside the scope of the ITS 2.0 loc-quality model. For the sake of conciseness and clarity, these are not listed in this document.)