W3C home > Mailing lists > Public > public-i18n-its-ig@w3.org > August 2016

Re: ITS rules for XLIFF 2.1

From: Sergey Nozhenko <sergey.nozhenko@logrusglobal.com>
Date: Tue, 23 Aug 2016 22:50:18 +0300
To: Felix Sasaki <fsasaki@w3.org>
CC: Serge Gladkoff <serge.gladkoff@gmail.com>, "public-i18n-its-ig@w3.org" <public-i18n-its-ig@w3.org>, Renat Bikmatov <renat.bikmatov@logrusglobal.com>
Message-ID: <25b85751-2305-06f0-50c3-c7dc3cee071a@logrusglobal.com>

sm and em elements may be nested in mrk and overlap it. For example:

<xliff version="2.0" xmlns="urn:oasis:names:tc:xliff:document:2.0" 
srcLang="en" xmlns:itsm="urn:oasis:names:tc:xliff:itsm:2.1">
  <file id="f1">
   <unit id="u1">
     <source><mrk id="m1" translate="no" type="term">Text1 <sm id="sm1" 
type="itsm:generic" itsm:taClassRef="http://example/ontology#Thing" 
     <source>Text4<em startRef="sm1"/> text5.</source>


On 23.08.2016 19:48, Felix Sasaki wrote:
> Apologies for the late reply, Sergey, Serge and all.
> The issue is an XLIFF issue related to the annotations mechanism
> http://docs.oasis-open.org/xliff/xliff-core/v2.0/os/xliff-core-v2.0-os.html#annotations
> if an annotation in XLIFF is represented with sm and em, the 
> application has to find the content relating to the annotation.
> I think this is doable, both for general XLIFF annotations (e.g. of 
> terms) and ITS annotations. I updated my implementation with an XPath 
> expression that has a larger search space than the previous one. The 
> new expression searches for the corresponding em tag in the following 
> nodes that have the same parent node type (e.g. all source elements or 
> all target elements).
> It seems to work, see the more complex input
> https://github.com/fsasaki/its20-extractor/blob/master/sample/xliff21sample/inputfile.xml
> and output
> https://github.com/fsasaki/its20-extractor/blob/master/sample/xliff21sample/output-inline-annotation.xml
> and the adapted XPath at
> https://github.com/fsasaki/its20-extractor/commit/62428b4484df7a073be3c2c0033e2a389dc83350
> in tools/datacategories-2-xsl.xsl.
> I’m happy to work on this more if you give me more XLIFF annotation 
> samples.
> Best,
> Felix
>> Am 18.08.2016 um 11:32 schrieb Sergey Nozhenko 
>> <sergey.nozhenko@logrusglobal.com 
>> <mailto:sergey.nozhenko@logrusglobal.com>>:
>> How about this:
>> <xliff version="2.0" xmlns="urn:oasis:names:tc:xliff:document:2.0" srcLang="en" trgLang="ru"
>>   xmlns:itsm="urn:oasis:names:tc:xliff:itsm:2.1">
>>   <file id="f1">
>>    <unit id="u1">
>>     <segment>
>>      <source><sm id="sm1" type="itsm:generic" itsm:taClassRef="http://nerd.eurecom.fr/ontology#Place"
>>       itsm:taIdentRef="http://dbpedia.org/resource/Arizona"/>Arizona</source>
>>      <target>Аризона</target>
>>     </segment>
>>     <segment>
>>      <source><em startRef="sm1"/> Yeah!</source>
>>      <target>Да!</target>
>>     </segment>
>>    </unit>
>>   </file>
>> </xliff>
>> Serge
>> *From:*Felix Sasaki <mailto:fsasaki@w3.org>
>> *Sent:*18 августа 2016 г. 8:18
>> *To:*Serge Gladkoff <mailto:serge.gladkoff@gmail.com>
>> *Cc:*public-i18n-its-ig@w3.org 
>> <mailto:public-i18n-its-ig@w3.org>;Renat Bikmatov 
>> <mailto:renat.bikmatov@logrusglobal.com>;Sergey Nozhenko 
>> <mailto:sergey.nozhenko@logrusglobal.com>
>> *Subject:*Re: ITS rules for XLIFF 2.1
>>> Am 17.08.2016 um 23:08 schrieb Serge Gladkoff 
>>> <serge.gladkoff@gmail.com <mailto:serge.gladkoff@gmail.com>>:
>>> Hello Felix,
>>> I am sorry to say this but our developers believe that this is a 
>>> clear case where ITS hit rock-bottom, so to speak.
>>> The function of <sm>/<em> tags is to markup the areas which cannot 
>>> be annotated by one tag because this would result in invalid XML 
>>> file. This happens when the markup is conflicting with other tags. 
>>> For example, with segmentation.
>>> In such cases inheritance does not work because the beginning of the 
>>> unit may find itself inside one tag, and the end – inside another, 
>>> and even on different levels.
>> Indeed - that was exactly my point.
>>> How one could describe ITS tags distribution in such cases?
>> By keeping your ITS processor (including inheritance behavior) as is, 
>> and then specify additional processing for sm, as defined below. My 
>> main point was that this does not change the behavior of a conformant 
>> ITS processor. It is *additional* behavior.
>>> Indeed, it is far from clear.
>>> I wouldn't call this “a small burden”.
>> I implemented this as an additional behavior of my ITS processor. See
>> https://github.com/fsasaki/its20-extractor/commit/4816b29f8b7010f307c5dad98b1ab4aa92c4ae70
>> the changes to datacategories-2-xsl.xsl . The changes was 4 lines of 
>> code. I am happy to look at your code with your developers, if that 
>> helps, to lower the burden.
>> Best,
>> Felix
>>> Regards,
>>> Serge
>>> *From:*Felix Sasaki [mailto:fsasaki@w3.org]
>>> *Sent:*Tuesday, August 16, 2016 7:20 PM
>>> *To:*public-i18n-its-ig@w3.org <mailto:public-i18n-its-ig@w3.org>
>>> *Subject:*ITS rules for XLIFF 2.1
>>> Hi all,
>>> in the OASIS TC, currently the support of ITS in XLIFF 2.1 is being 
>>> discussed.
>>> As part of the discussion an ITS rules file is developed. The file 
>>> should allow general ITS processors to work with XLIFF 2.X 
>>> documents. There is one issue: XLIFF has elements „sm“ and „em“ 
>>> which are empty markers. (ITS or any other) information then relates 
>>> to the content between the start and end marker.
>>> Below is a mail I had sent to the XLIFF list to find a work around. 
>>> This would put a (small) burden on ITS processors, to deal with the 
>>> sm / em elements. See below, I tried this with my general XSLT 
>>> implementation. What do people think on this, esp. implementers?
>>> Best,
>>> Felix
>>> Anfang der weitergeleiteten Nachricht:
>>> *Von: *Felix Sasaki <felix@sasakiatcf.com <mailto:felix@sasakiatcf.com>>
>>> *Betreff: Implementation of XLIFF 2.1 - ITS module*
>>> *Datum: *12. August 2016 um 11:51:14 MESZ
>>> *An: *XLIFF Main List <xliff@lists.oasis-open.org 
>>> <mailto:xliff@lists.oasis-open.org>>
>>> Hi all,
>>> I started an ITS module implementation relying on my generic ITS 
>>> processor. See the processed files here
>>> https://github.com/fsasaki/its20-extractor/tree/master/sample/xliff21sample
>>> external-rules.xml contains the rules, currently only for text 
>>> analytics. inputfile.xml is an XLIFF 2.1 input file, currently with 
>>> ITS Text Analytics information. The output is as a list of XPath 
>>> expressions in nodelist-with-its-information.xml and as inline 
>>> annotations in output-inline-annotation.xml
>>> The output shows one issue which we had discussed before, see below, 
>>> taken from output-inline-annotation.xml
>>> <source>
>>>                 <itsAnn xmlns=""/>
>>>                 <sm id="sm1"
>>>                     type="itsm:generic"
>>>                     itsm:taClassRef="http://nerd.eurecom.fr/ontology#Place"
>>>                     itsm:taIdentRef="http://dbpedia.org/resource/Arizona">
>>>                    <itsAnn xmlns="">
>>>                       <elem>
>>>                          <taClassRefPointer xmlns:xlf2="urn:oasis:names:tc:xliff:document:2.0"
>>>                                             xmlns:its="http://www.w3.org/2005/11/its"
>>>                                             xmlns:datc="http://example.com/datacats"
>>>                                             itsm:taClassRef="http://nerd.eurecom.fr/ontology#Place"/>
>>>                          <taIdentRefPointer xmlns:xlf2="urn:oasis:names:tc:xliff:document:2.0"
>>>                                             xmlns:its="http://www.w3.org/2005/11/its"
>>>                                             xmlns:datc="http://example.com/datacats"
>>>                                             itsm:taIdentRef="http://dbpedia.org/resource/Arizona"/>
>>>                       </elem>
>>>                    </itsAnn>
>>>                 </sm>Arizona<em startRef="sm1">
>>>                    <itsAnn xmlns=""/>
>>>                 </em>
>>>              </source>
>>>  With the ITS rules file, „sm“ is annotated to have the text 
>>> analytics information. But it is actually the content between sm and 
>>> em that should be annotated. I don’t know how to resolve this. Maybe 
>>> we should add to the ITS module the constraint that extends general 
>>> ITS processors: if the selected element is XLIFF sm, apply the ITS 
>>> information to the next em which corresponds to sm, via the startRef 
>>> attribute. This would be a small burden on the ITS processors, but 
>>> would greatly simply the creation of the ITS/XLIFF rules file.
>>> Thoughts?
>>> Best,
>>> Felix
Received on Tuesday, 23 August 2016 19:51:43 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 23 August 2016 19:51:46 UTC