Re: Surrogate Code Points in Tests?

Yes, I agree with option 3 AND option 2.

--Peter

From: Alex Milowski 
Sent: Friday, May 17, 2013 5:17 PM
To: Peter Occil 
Cc: Richard Cyganiak ; Eric Prud'hommeaux ; public-rdf-comments@w3.org 
Subject: Re: Surrogate Code Points in Tests?

It would be great to mark these tests with something to indicate these concerns.  Option (3) is really a solution you can implement.



On Fri, May 17, 2013 at 2:13 PM, Peter Occil <poccil14@gmail.com> wrote:

  I believe the point of what Alex is saying is that an implementation in
  JavaScript that represents Turtle documents as strings ought to take care to interpret 
  Unicode characters in strings correctly, since Unicode supports many more 
  characters than the 16-bit code units in JavaScript strings can represent. 
  For example, the charCodeAt method in JavaScript returns only the value 
  of the 16-bit code unit in the given index of the string, not necessarily
  the Unicode character found there; and therefore can return a surrogate code point.

  This applies not only to JavaScript, but also Java, .NET, and any other environment 
  whose strings consist of 16-bit Unicode code units.

  Accordingly, I believe this issue can be resolved by option 2.

  --Peter

  From: Richard Cyganiak 
  Sent: Friday, May 17, 2013 3:45 PM
  To: Alex Milowski 
  Cc: Eric Prud'hommeaux ; public-rdf-comments@w3.org 
  Subject: Re: Surrogate Code Points in Tests?

  I understand only half of what's being discussed in this thread. Maybe this is relevant: RDF Concepts says that IRIs and literals SHOULD be in Unicode normal form C.

  Apologies if that's not relevant here.

  Best,
  Richard


  On 17 May 2013, at 19:17, Alex Milowski <alex@milowski.com> wrote:


    The attached file (UTF-8 version) is essentially the same as the one I got from the test suite with just slightly different line endings.   

    The problem here is that U+10000 and U+EFFFF are not representable in UCS-2 and it appears that most browsers are using UCS-2 at some level for the strings in Javascript.  As such, these code point are turned into surrogate pairs (U+D800 + U+DC00 and U+DB7F + U+DFFF) which aren't actually part of UCS-2 as legal code points.  The surrogate characters are also not allowed by the productions in prefixes and other such names. 

    The test is unusable in an environment, like most current browsers, that produce surrogate pairs from the UTF-8 encoding.  That is, unusable without some enhanced level of support for surrogates.

    There are a number of options to resolve this:

       1. The test will just fail for such environments without special support for surrogates.
       2. We mark this test as requiring surrogate / UTF-16 handling.  Passing the test is quality of implementation question.
       3. Such environments are expected to check for pairs of surrogate values in place of the code points [#x10000-#xEFFFF] even though the resulting identifier is unlikely to be generally usable.

    Option (1) means I just fail the test and move on. I'm not particularly partial to this.

    Option (2) allows greater diversity of processor environments but certainly runs counter to promoting proper Unicode support.  It would be good to have a variant that tests only those in the Basic Multilingual Plane.

    Option (3) allows a processor to pass and punts the use of the code point back to the user.  In the case of these particular code points and identifiers, the user of the result will have similar issues when inspecting or using values from the processor.  This will be no different for data received from other places that has the same code points as Javascript will treat them all the same and produce surrogate pairs.

    There absolutely needs to be much more documentation attached to this test.  The identifiers (e.g. prefix) are constructed by taking the first and last character of each range in the PN_CHARS_BASE production.  I have to admit I really didn't recognize that until much later in my research into why this test didn't pass.  In addition, the general issue of UCS-2  vs UTF-16 representations and surrogates needs to be minimally referenced in the description.

    It would be good for everyone else to avoid a lot the research I just did with a well-written test description. 

    BTW, there's a good write up of Javascript's issues at [1]

    [1] http://mathiasbynens.be/notes/javascript-encoding



    On Fri, May 17, 2013 at 4:37 AM, Eric Prud'hommeaux <eric@w3.org> wrote:

      * Alex Milowski <alex@milowski.com> [2013-05-16 23:44-0700]

      > In looking at test:
      >
      >    prefix_with_PN_CHARS_BASE_character_boundaries.ttl  [1]
      >
      > There are the code points u+dc00, u+db7f, and u+dfff in the last part of
      > the prefix.  The code points u+d800-u+dfff are not valid unicode characters.
      >
      > Why are these in a positive test?
      >
      > [1]
      > https://dvcs.w3.org/hg/rdf/raw-file/default/rdf-turtle/tests-ttl/prefix_with_PN_CHARS_BASE_character_boundaries.ttl


      In the prefix, I see these codepoints: u+41 u+5a u+61 u+7a u+c0 u+d6
      u+d8 u+f6 u+f8 u+2ff u+370 u+37d u+37f u+1fff u+200c u+200d u+2070
      u+218f u+2c00 u+2fef u+3001 u+d7ff u+f900 u+fdcf u+fdf0 u+fffd u+10000
      u+effff .

      I've attached two variants the test, one encoded in UTF-8, which is
      legal turtle and has only the codepoints above, and UTF-16, which uses
      surrogate to encode the codepoints u+10000 and u+effff . Is it
      possible that your buffer got re-encoded as UTF-16 at some point?

      If this message resolves your comment, please reply with "[RESOLVED]"
      in the subject.


      > --
      > --Alex Milowski
      > "The excellence of grammar as a guide is proportional to the paucity of the
      > inflexions, i.e. to the degree of analysis effected by the language
      > considered."
      >
      > Bertrand Russell in a footnote of Principles of Mathematics


      --
      -ericP





    -- 
    --Alex Milowski
    "The excellence of grammar as a guide is proportional to the paucity of the
    inflexions, i.e. to the degree of analysis effected by the language
    considered."

    Bertrand Russell in a footnote of Principles of Mathematics 




-- 
--Alex Milowski
"The excellence of grammar as a guide is proportional to the paucity of the
inflexions, i.e. to the degree of analysis effected by the language
considered."

Bertrand Russell in a footnote of Principles of Mathematics