RE: Character Encoding Question

John, Tom, others,

I don't know if this helps or hurts, but my original comment stemmed from my
understanding that UCS-2 != Unicode. From RFC-2279 (which is actually the
UTF-8 RFC, but has this nice description), F. Yergeau:

"It is noteworthy that the same set of characters [UCS-2, current BMP] is
defined by the Unicode standard [UNICODE], which further defines additional
character properties and other application details of great interest to
implementors, but does not have the UCS-4 encoding.  Up to the present time,
changes in Unicode and amendments to ISO/IEC 10646 have tracked each other,
so that the character repertoires and code point assignments have remained
in sync."

My understanding is that for character point assignment, UCS-2 == Unicode
evaluates TRUE and will continue to evaluate TRUE. UCS-4 == Unicode will
always evaluate FALSE. "UCS" is probably the term desired in the Canonical
XML Version 1.0 specification. The fact that most implementors will use
Unicode to generate UCS character point data appears to be an implementation
detail. By using UCS as the specifier for character data, when characters
are added to the BMP and when those characters are outside of UCS-2/Unicode
and in UCS-4, the specification will continue to have meaning.

Thanks,

Jeff

-----Original Message-----
From: John Boyer [mailto:jboyer@PureEdge.com]
Sent: Tuesday, November 28, 2000 5:20 PM
To: Tom Gindin
Cc: Martin J. Duerst; w3c-ietf-xmldsig@w3.org
Subject: RE: Character Encoding Question


Hi Tom,

UTF-8 and UTF-16 are both encodings of UCS, whether UCS-2 or UCS-4.  If I
understand correctly, UCS-n is a character domain used during processing,
and UTF-n is used for input and output.

I am certain that UTF-8 character sequences can encode UCS-4, and I know
that UCS-2 is a two byte per char character domain for use in processing
scenarios where only the BMP is required.  This would be all scenarios right
now (according to the Unicode 3.0 manual) because nothing is yet defined
outside of the BMP, although ISO 10646-2 is likely to change that (again,
according to the Unicode 3.0 manual).

One thing I don't know for sure is whether Unicode == UCS-2?

If so, then our current sentence is certain wrong because I'm sure we don't
mean that NFC should be applied to UTF-8 and UTF-16 encodings of UCS-n.

If Unicode != UCS-2, then A) what's the difference, and B) it would be
helpful if someone would confirm whether UCS-n is ever used for
transportation of character data, or whether this is done solely by UTF-n
formats.  If so, will it continue in this fashion in the future?

Finally, I don't know whether anyone reads or writes UCS-n data directly,
but I do know that our intent was that UTF-n data would not have NFC applied
to it.

Thanks,
John Boyer
Team Leader, Software Development
Distributed Processing and XML
PureEdge Solutions Inc.
Creating Binding E-Commerce
v: 250-479-8334, ext. 143  f: 250-479-3772
1-888-517-2675   http://www.PureEdge.com <http://www.pureedge.com/>



-----Original Message-----
From: w3c-ietf-xmldsig-request@w3.org
[mailto:w3c-ietf-xmldsig-request@w3.org]On Behalf Of Tom Gindin
Sent: Tuesday, November 28, 2000 4:53 PM
To: John Boyer
Cc: Martin J. Duerst; w3c-ietf-xmldsig@w3.org
Subject: Re: Character Encoding Question



     Is what is meant "... from an encoding which is neither a UCS-n
encoding nor a UTF-n encoding"?  That would seem to cover UCS-2, UCS-4,
UTF-8, and UTF-16 (along with UTF-7 for good measure).  If UTF-8 is not
included, although the NFC transformation would seem to have no effect on
it, just replace "UTF-n" by "UTF-16" in the sentence above.

          Tom Gindin

"John Boyer" <jboyer@PureEdge.com>@w3.org on 11/28/2000 05:39:27 PM

Sent by:  w3c-ietf-xmldsig-request@w3.org


To:   "Martin J. Duerst" <duerst@w3.org>, <w3c-ietf-xmldsig@w3.org>
cc:
Subject:  Character Encoding Question



Hi Martin and group,

I received a letter today from Jeff Cochran (JCochran@docutouch.com)
regarding a tweak that would appear to be needed regarding c14n and xml
signature.

The I18N group asked us to include a sentence along the lines of "REQUIRED
to use Normalization Form C [NFC] when converting an XML document to the
UCS
character domain from a non-Unicode encoding".

Apparently this is not exactly what is meant since UCS-4 character planes
outside of the BMP are technically non-Unicode.  The point Jeff makes is
that he doesn't know whether to apply NFC to UCS data that appears outside
of the BMP.

Question:  Should the statement be rewritten?  If so, how?

Thanks,
John Boyer
Team Leader, Software Development
Distributed Processing and XML
PureEdge Solutions Inc.
Creating Binding E-Commerce
v: 250-479-8334, ext. 143  f: 250-479-3772
1-888-517-2675   http://www.PureEdge.com <http://www.pureedge.com/>



-----Original Message-----
From: w3c-ietf-xmldsig-request@w3.org
[mailto:w3c-ietf-xmldsig-request@w3.org]On Behalf Of Martin J. Duerst
Sent: Friday, November 24, 2000 6:17 PM
To: w3c-ietf-xmldsig@w3.org
Cc: lilley@w3.org
Subject: Fwd: I18N problem in XML canonicalisation


Chris Lilley just pointed out the following problem
in C14N. I think this at least has to be explained
much more clearly in the notes.

>http://www.w3.org/TR/xml-c14n#Example-UTF8
>
>Demonstrates using *two* NCRs foa single UTF-8 character (because it uses
>two bytes in UTF8 !!!

It's not really NCRs. It's a special notation to stand in for byte values.


>I suspect you may have a problem with that..... given that even surrogates
>use a single NCR not two. Also, its not clear the result is even
>wellformed!

There needs to be a much better note to make very clear that (different
to the other examples), this example is not really intended to be XML
and cannot be used directly in a test. It would also be advisable
to provide an actual file that contains the real bytes, or to point
to it if that's already around.

Regards,    Martin.