Re: claimed completion on "ACTION-233: Publish the consolidated test suite"

On 21/03/13 03:51, Eric Prud'hommeaux wrote:
> RDF and I18N folks, we have an interesting situation where we permit
> U+F900-U+FA0D to appear in local names, but advise against anything
> which is not NFC. So, what do we test?

The grammar is wider than the acceptable URIs in several places - it's 
inevitable.  We're expecting URI checking to be done after parsing in a 
very strict implementation.

So test good practice and recognize that not everywhere is completely 
up-to-date on everything.


 Andy

> Everything a Turtle parser
> could encouter? Currently assigned characters that are in NFC?
> Identifiers consisting of a single letter 'a', under the assumption
> that all others will work by extension?
>
>
> * Gavin Carothers <gavin@carothers.name> [2013-03-20 13:17-0700]
>> http://www.unicode.org/charts/PDF/U1F00.pdf U+1FFF is not a character.
>> http://www.unicode.org/charts/PDF/U2150.pdf U+218F is not a character.
>> No chart for code point U+2FEF could be located. Most likely this is
>> because no character is assigned to this code point yet.
>> http://www.unicode.org/charts/PDF/UD7B0.pdf U+D7FF is not a character.
>> http://www.unicode.org/charts/PDF/UFB50.pdf U+FDCF is not a character.
>> No chart for code point U+EFFFF could be located. Most likely this is
>> because no character is assigned to this code point yet
>>
>>
>> New string based on the above missing characters tested in Python 3.3
>> (earlier versions of python not supported, only one with Unicode 6.1.0)
>
> I banged briefly on finding an ubuntu package for Python 3.3
> (currently at 3.2). Ended up with something called perl. sigh.
>
> use Unicode::Normalize;
> $s = "AZaz\x{00c0}\x{00d6}\x{00d8}\x{00f6}\x{00f8}\x{02ff}\x{0370}\x{037d}\x{0384}\x{1ffe}\x{200c}\x{200d}\x{2070}\x{217f}\x{2c00}\x{2fcf}\x{3001}\x{d7fb}\x{f900}\x{fdc7}\x{fdf0}\x{fffd}\x{00010000}\x{0001f52b}";
> p $s cmp NFC($s);
> => 1 -- strings are different. so now to look for the first candidate:
>
> for (0xf900..0xfdcf) {
>      if (ord(Unicode::Normalize::NFC(chr($_))) == $_) {
>          printf("%x\n", $_);
>          last;
>      }
> }
> => fa0e
>
> # checked with
> $s = "AZaz\x{00c0}\x{00d6}\x{00d8}\x{00f6}\x{00f8}\x{02ff}\x{0370}\x{037d}\x{0384}\x{1ffe}\x{200c}\x{200d}\x{2070}\x{217f}\x{2c00}\x{2fcf}\x{3001}\x{d7fb}\x{fa0e}\x{fdc7}\x{fdf0}\x{fffd}\x{00010000}\x{0001f52b}";
> p $s cmp NFC($s);
> => 0 -- equivalent
>
> The currently unassigned characters don't impact NFC:
> $s = "AZaz\x{00c0}\x{00d6}\x{00d8}\x{00f6}\x{00f8}\x{02ff}\x{0370}\x{037d}\x{037f}\x{1fff}\x{200c}\x{200d}\x{2070}\x{218f}\x{2c00}\x{2fef}\x{3001}\x{d7ff}\x{fa0e}\x{fdcf}\x{fdf0}\x{fffd}\x{10000}\x{effff}"
> p $s cmp NFC($s);
> => 0 -- equivalent
>
>
>> import unicodedata
>> s =
>> "AZaz\u00c0\u00d6\u00d8\u00f6\u00f8\u02ff\u0370\u037d\u0384\u1ffe\u200c\u200d\u2070\u217f\u2c00\u2fcf\u3001\ud7fb\uf900\ufdc7\ufdf0\ufffd\U00010000\U0001f52b"
>>
>> def display_string(s):
>> for c in s:
>> print("""Character: {c!s}
>> Codepoint: {code:x}
>> Name: {name}
>> Combining: {combining}
>> """.format(
>> c=c,
>> code=ord(c),
>> name=unicodedata.name(c),
>> combining=unicodedata.combining(c),
>> ))
>>
>> n = unicodedata.normalize("NFC", s)
>>
>> display_string(s)
>> print("\n ------------------ \n ")
>> display_string(n)
>>
>> assert n == s
>>
>> Yeah, they aren't the same. The offending character is f900:
>>
>> CJK COMPATIBILITY IDEOGRAPH-F900 which in normal form is CJK UNIFIED
>> IDEOGRAPH-8C48
>>
>> Finding something in the F900ish range is left to Eric. Script above can be
>> modified until it passes.
>>
>> Cheers,
>> Gavin
>>
>>
>>
>>
>>
>> On Wed, Mar 20, 2013 at 12:13 PM, Eric Prud'hommeaux <eric@w3.org> wrote:
>>
>>> * Andy Seaborne <andy.seaborne@epimorphics.com> [2013-03-20 17:36+0000]
>>>> The TTL has U+037E but ...
>>>>
>>>> PN_CHARS_BASE has a hole specifically for that
>>>>
>>>> [#x0370-#x037D] | [#x037F-#x1FFF]
>>>>
>>>> => not a legal char.
>>>
>>> Yeah, I screwed that up. I should have gone the other way 'cause it's at
>>> the bottom of a range (unlike all the other unassigned chars). Attached are
>>> the same tests with s/37f/384/. Could you chop off after the "AZaz" and see
>>> if that works and do a binary search to see what it's complaining about?
>>>
>>> I18N folks, could you tell me why an NFC validator is objecting to this
>>> (beautiful) IRI and if there's some validator I can use for testing:?
>>>    <http://a.example/AZazÀÖØöø˿Ͱͽ΄῾‌‍⁰↉Ⰰ⿕、ퟻ豈ﷇﷰ�𐀀>
>>> The goal is to test as much as possible the valid input to <
>>> http://www.w3.org/TR/turtle/#grammar-production-PrefixedName>. In turtle,
>>> the localName gets appended to the namespace, hence the url above. The
>>>
>>>    [163s] PN_CHARS_BASE ::=    [A-Z] | [a-z] | [#x00C0-#x00D6] |
>>> [#x00D8-#x00F6] | [#x00F8-#x02FF] | [#x0370-#x037D] | [#x037F-#x1FFF] |
>>> [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] |
>>> [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
>>>
>>> production is taken from <http://www.w3.org/TR/REC-xml/#NT-NameStartChar>:
>>>
>>>    [4] NameStartChar ::= ":" | [A-Z] | "_" | [a-z] | [#xC0-#xD6] |
>>> [#xD8-#xF6] | [#xF8-#x2FF] | [#x370-#x37D] | [#x37F-#x1FFF] |
>>> [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] |
>>> [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
>>>
>>>
>>>
>>>> Removing it (Greek question mark), I then get:
>>>>
>>>> WARN  [line: 2, col: 43] Bad IRI:
>>>> <http://a.example/AZaz???????????????????????> Code: 46/NOT_NFC in
>>>> PATH: The IRI is not in Unicode Normal Form C.
>>>> WARN  [line: 2, col: 43] Bad IRI:
>>>> <http://a.example/AZaz???????????????????????> Code: 47/NOT_NFKC in
>>>> PATH: The IRI is not in Unicode Normal Form KC.
>>>> WARN  [line: 2, col: 43] Bad IRI:
>>>> <http://a.example/AZaz???????????????????????> Code:
>>>> 56/COMPATIBILITY_CHARACTER in PATH: TODO
>>>>
>>>> with or without the last char.
>>>>
>>>>> I poked around looking for composing characters in the PN_CHARS_BASE
>>>>> character ranges. \u02ff MODIFIER LETTER LOW LEFT ARROW seemed like it
>>>>> could be a culprit, but fileformat.info claims it's not in a combining
>>>>> class. Likewise \ufffd REPLACEMENT CHARACTER
>>>>>
>>>>> There are a bunch of yet-unassigned characters which could be confusing
>>>>> a vigilent IRI checkr. I've mapped those to the highest currently-
>>>>> assigned characters in their respective range (per fileformat.info):
>>>>>
>>>>>      \u037f   37e
>>>>>      \u1fff  1ffe
>>>>>      \u218f  2189
>>>>>      \u2fef  2fd5
>>>>>      \ud7ff  d7fb
>>>>>      \ufdcf  fdc7
>>>>> \U000effff e01ef
>>>>>
>>>>> attached is a variant of
>>>>>    localName_with_PN_CHARS_BASE_character_boundaries.{nt,ttl}
>>>>> with the values substituted. (I pass this modified test so there
>>>>> shouldn't be any typos in it.) If it still doesn't work, try chopping
>>>>> off the last character 'cause it's a variation selector which ostensibly
>>>>> is NF{,K}{C,D} valid, but may not have been when jjc wrote your checker.
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> -ericP
>>>
>>>
>

Received on Thursday, 21 March 2013 10:49:19 UTC