RE: ITS rules for XLIFF 2.1

Re-sending just in case the list wouldn't accept contributions from @logrusglobal.com alias.

 

 

From: Sergey Nozhenko [mailto:sergey.nozhenko@logrusglobal.com] 
Sent: Tuesday, August 23, 2016 10:50 PM
To: Felix Sasaki
Cc: Serge Gladkoff; public-i18n-its-ig@w3.org; Renat Bikmatov
Subject: Re: ITS rules for XLIFF 2.1

 

Hi,

sm and em elements may be nested in mrk and overlap it. For example:

<xliff version="2.0" xmlns="urn:oasis:names:tc:xliff:document:2.0" srcLang="en" xmlns:itsm="urn:oasis:names:tc:xliff:itsm:2.1">
 <file id="f1">
  <unit id="u1">
   <segment>
    <source><mrk id="m1" translate="no" type="term">Text1 <sm id="sm1" type="itsm:generic" itsm:taClassRef= <http://example/ontology#Thing> "http://example/ontology#Thing" itsm:taIdentRef= <http://example.com/ref> "http://example.com/ref"/>Text2.</mrk></source>
   </segment>
   <segment>
    <source>Text4<em startRef="sm1"/> text5.</source>
   </segment>
  </unit>
 </file>
</xliff>

Serge
 

On 23.08.2016 19:48, Felix Sasaki wrote:

Apologies for the late reply, Sergey, Serge and all.  

 

The issue is an XLIFF issue related to the annotations mechanism

http://docs.oasis-open.org/xliff/xliff-core/v2.0/os/xliff-core-v2.0-os.html#annotations

if an annotation in XLIFF is represented with sm and em, the application has to find the content relating to the annotation.

 

I think this is doable, both for general XLIFF annotations (e.g. of terms) and ITS annotations. I updated my implementation with an XPath expression that has a larger search space than the previous one. The new expression searches for the corresponding em tag in the following nodes that have the same parent node type (e.g. all source elements or all target elements).

 

It seems to work, see the more complex input

https://github.com/fsasaki/its20-extractor/blob/master/sample/xliff21sample/inputfile.xml

and output

https://github.com/fsasaki/its20-extractor/blob/master/sample/xliff21sample/output-inline-annotation.xml

and the adapted XPath at

https://github.com/fsasaki/its20-extractor/commit/62428b4484df7a073be3c2c0033e2a389dc83350

in tools/datacategories-2-xsl.xsl.

 

I’m happy to work on this more if you give me more XLIFF annotation samples.

 

Best,

 

Felix

 

 

Am 18.08.2016 um 11:32 schrieb Sergey Nozhenko <sergey.nozhenko@logrusglobal.com>:

 

How about this:

 

<xliff version="2.0" xmlns="urn:oasis:names:tc:xliff:document:2.0" srcLang="en" trgLang="ru"
 xmlns:itsm="urn:oasis:names:tc:xliff:itsm:2.1">
 <file id="f1">
  <unit id="u1">
   <segment>
    <source><sm id="sm1" type="itsm:generic" itsm:taClassRef=" <http://nerd.eurecom.fr/ontology#Place> http://nerd.eurecom.fr/ontology#Place"
     itsm:taIdentRef=" <http://dbpedia.org/resource/Arizona> http://dbpedia.org/resource/Arizona"/>Arizona</source>
    <target>Аризона</target>
   </segment>
   <segment>
    <source><em startRef="sm1"/> Yeah!</source>
    <target>Да!</target>
   </segment>
  </unit>
 </file>

</xliff>

 

Serge

 

From:  <mailto:fsasaki@w3.org> Felix Sasaki
Sent: 18 августа 2016 г. 8:18
To:  <mailto:serge.gladkoff@gmail.com> Serge Gladkoff
Cc:  <mailto:public-i18n-its-ig@w3.org> public-i18n-its-ig@w3.org;  <mailto:renat.bikmatov@logrusglobal.com> Renat Bikmatov;  <mailto:sergey.nozhenko@logrusglobal.com> Sergey Nozhenko
Subject: Re: ITS rules for XLIFF 2.1

 

 

Am 17.08.2016 um 23:08 schrieb Serge Gladkoff < <mailto:serge.gladkoff@gmail.com> serge.gladkoff@gmail.com>:

 

Hello Felix,

 

I am sorry to say this but our developers believe that this is a clear case where ITS hit rock-bottom, so to speak.

 

The function of <sm>/<em> tags is to markup the areas which cannot be annotated by one tag because this would result in invalid XML file. This happens when the markup is conflicting with other tags. For example, with segmentation. 

 

In such cases inheritance does not work because the beginning of the unit may find itself inside one tag, and the end – inside another, and even on different levels.

 

Indeed - that was exactly my point. 





 

How one could describe ITS tags distribution in such cases?

 

By keeping your ITS processor (including inheritance behavior) as is, and then specify additional processing for sm, as defined below. My main point was that this does not change the behavior of a conformant ITS processor. It is *additional* behavior. 





Indeed, it is far from clear.

 

I wouldn't call this “a small burden”.

 

I implemented this as an additional behavior of my ITS processor. See 

 <https://github.com/fsasaki/its20-extractor/commit/4816b29f8b7010f307c5dad98b1ab4aa92c4ae70> https://github.com/fsasaki/its20-extractor/commit/4816b29f8b7010f307c5dad98b1ab4aa92c4ae70

the changes to datacategories-2-xsl.xsl . The changes was 4 lines of code. I am happy to look at your code with your developers, if that helps, to lower the burden.

 

Best,

 

Felix





 

Regards,

Serge

 

 

 

From: Felix Sasaki [ <mailto:fsasaki@w3.org> mailto:fsasaki@w3.org] 
Sent: Tuesday, August 16, 2016 7:20 PM
To: public-i18n-its-ig@w3.org
Subject: ITS rules for XLIFF 2.1

 

Hi all,

 

in the OASIS TC, currently the support of ITS in XLIFF 2.1 is being discussed.

 

As part of the discussion an ITS rules file is developed. The file should allow general ITS processors to work with XLIFF 2.X documents. There is one issue: XLIFF has elements „sm“ and „em“ which are empty markers. (ITS or any other) information then relates to the content between the start and end marker.

 

Below is a mail I had sent to the XLIFF list to find a work around. This would put a (small) burden on ITS processors, to deal with the sm / em elements. See below, I tried this with my general XSLT implementation. What do people think on this, esp. implementers?

 

Best,

 

Felix 

 

 

 

Anfang der weitergeleiteten Nachricht:

 

Von: Felix Sasaki < <mailto:felix@sasakiatcf.com> felix@sasakiatcf.com>

Betreff: Implementation of XLIFF 2.1 - ITS module

Datum: 12. August 2016 um 11:51:14 MESZ

An: XLIFF Main List < <mailto:xliff@lists.oasis-open.org> xliff@lists.oasis-open.org>

 

Hi all,

 

I started an ITS module implementation relying on my generic ITS processor. See the processed files here

 <https://github.com/fsasaki/its20-extractor/tree/master/sample/xliff21sample> https://github.com/fsasaki/its20-extractor/tree/master/sample/xliff21sample

external-rules.xml contains the rules, currently only for text analytics. inputfile.xml is an XLIFF 2.1 input file, currently with ITS Text Analytics information. The output is as a list of XPath expressions in nodelist-with-its-information.xml and as inline annotations in output-inline-annotation.xml

 

The output shows one issue which we had discussed before, see below, taken from output-inline-annotation.xml

 

<source>
               <itsAnn xmlns=""/>
               <sm id="sm1"
                   type="itsm:generic"
                   itsm:taClassRef=" <http://nerd.eurecom.fr/ontology#Place> http://nerd.eurecom.fr/ontology#Place"
                   itsm:taIdentRef=" <http://dbpedia.org/resource/Arizona> http://dbpedia.org/resource/Arizona">
                  <itsAnn xmlns="">
                     <elem>
                        <taClassRefPointer xmlns:xlf2="urn:oasis:names:tc:xliff:document:2.0"
                                           xmlns:its=" <http://www.w3.org/2005/11/its> http://www.w3.org/2005/11/its"
                                           xmlns:datc=" <http://example.com/datacats> http://example.com/datacats"
                                           itsm:taClassRef=" <http://nerd.eurecom.fr/ontology#Place> http://nerd.eurecom.fr/ontology#Place"/>
                        <taIdentRefPointer xmlns:xlf2="urn:oasis:names:tc:xliff:document:2.0"
                                           xmlns:its=" <http://www.w3.org/2005/11/its> http://www.w3.org/2005/11/its"
                                           xmlns:datc=" <http://example.com/datacats> http://example.com/datacats"
                                           itsm:taIdentRef=" <http://dbpedia.org/resource/Arizona> http://dbpedia.org/resource/Arizona"/>
                     </elem>
                  </itsAnn>
               </sm>Arizona<em startRef="sm1">
                  <itsAnn xmlns=""/>
               </em>
            </source>

 

 With the ITS rules file, „sm“ is annotated to have the text analytics information. But it is actually the content between sm and em that should be annotated. I don’t know how to resolve this. Maybe we should add to the ITS module the constraint that extends general ITS processors: if the selected element is XLIFF sm, apply the ITS information to the next em which corresponds to sm, via the startRef attribute. This would be a small burden on the ITS processors, but would greatly simply the creation of the ITS/XLIFF rules file. 

 

Thoughts?

 

Best,

 

Felix