Re: ISO 8859 -8:1999

Whoo -- I sent this on Dec 5th.

Mark
—————

Πόλλ’ ἠπίστατο ἔργα, κακῶς δ’ ἠπίστατο πάντα — Ὁμήρου Μαργίτῃ
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

  ----- Original Message -----
  From: Mark Davis
  To: Martin Duerst ; Francois Yergeau ; ietf-charsets@iana.org ; Keld J n
Simonsen ; Jonathan Rosenne
  Sent: Wednesday, December 05, 2001 07:09
  Subject: Re: ISO 8859 -8:1999


  Precisely. As discussed in TR22, 1.2.1:

  There may be cases where a specified character mapping table is not
available. In such cases, a best-fit mapping table can be used. However,
this technique should be used with caution, since otherwise data can be
corrupted. For example, in XML there are different strategies depending on
whether the process is parsing or generating.
    Suppose that you have two sets X and SUB_X, where X is a superset of
SUB_X. (That is, every roundtrip mapping that is in SUB_X is also in X, and
X may contain additional round-trip mappings.) Then:

      a.. It is ok to parse with X when the file is tagged as SUB_X. Since X
is a superset, all the characters will be read correctly. Any characters
that are not in SUB_X will be encoded as NCRs (e.g. ꯍ), and will
work.
      b.. It is ok to generate the file with SUB_X, and tag the file as X.
As long as you convert the characters that are not in SUB_X into NCRs,
everything works.
      c.. What is NOT ok is to parse with SUB_X when the file is tagged with
X — characters will be corrupted.
      d.. What is NOT ok is to generate the file with X, and tag the file
with SUB_X — characters will be corrupted.

  TR22: http://www.unicode.org/unicode/reports/tr22/#Illegal_and_Unassigned
  —————

  Ὀλίγοι ἔμφονες πολλῶν ἀφρόνων φοβερώτεροι — Πλάτωνος
  [For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

  http://www.macchiato.com
  ----- Original Message -----
  From: "Martin Duerst" <duerst@w3.org>
  To: "Francois Yergeau" <FYergeau@alis.com>; <ietf-charsets@iana.org>;
"Keld J n Simonsen" <keld@dkuug.dk>; "Jonathan Rosenne" <rosenne@qsm.co.il>
  Sent: Tuesday, December 04, 2001 21:27
  Subject: RE: ISO 8859 -8:1999


  > Hello Francois,
  >
  > I'm not exactly sure your argument with UTF-8 works.
  >
  > The problem becomes obvious if we assume that the UCS
  > is the reference. In this case, additions to the UCS
  > can always be made to work as good as the current
  > implementation (because we don't expect conversion
  > to legacy encodings to work, except for something
  > like NCRs in HTML/XML). However, additions to legacy
  > encodings will break interoperability if new data
  > is sent to old implementations with the old label,
  > in particular if the implementation converts to
  > UCS internally (which more and more implementations
  > do nowadays).
  >
  > Regards,   Martin.
  >
  > At 14:15 01/12/02 -0500, Francois Yergeau wrote:
  > >Keld Jn Simonsen wrote:
  > > > Jonathan Rosenne wrote:
  > > >> Justification: ISO_8859-8:1999 is a superset of ISO_8859-8:1988.
Valid
  > > >> ISO_8859-8:1988 data will still be valid under ISO_8859-8:1999. The
new
  > > >> characters were reserved in ISO_8859-8:1988. Registering
ISO_8859-8:1999
  > > >> as a separate character set would cause too much unnecessary
confusion.
  > > >
  > > >As the two charsets are not exactly equivalent, they should not be
  > > >aliases.
  > >
  > >Not so fast, please.  This question was debated at length, on this very
  > >list, when RFC 2279 was under scrutiny.  The result was that the label
  > >"UTF-8" was determined to be appropriate for all version of Unicode
after
  > >1.1, provided no incompatible change occurs, which is the same as
saying
  > >that subsequent versions have a superset relationship to previous ones.
The
  > >arguments are in the RFC itself, section 5:
  > >
  > >    It is noteworthy that the label "UTF-8" does not contain a version
  > >    identification, referring generically to ISO/IEC 10646.  This is
  > >    intentional, the rationale being as follows:
  > >
  > >    A MIME charset label is designed to give just the information
needed
  > >    to interpret a sequence of bytes received on the wire into a
sequence
  > >    of characters, nothing more (see RFC 2045, section 2.2, in [MIME]).
  > >    As long as a character set standard does not change incompatibly,
  > >    version numbers serve no purpose, because one gains nothing by
  > >    learning from the tag that newly assigned characters may be
received
  > >    that one doesn't know about.  The tag itself doesn't teach anything
  > >    about the new characters, which are going to be received anyway.
  > >
  > >    Hence, as long as the standards evolve compatibly, the apparent
  > >    advantage of having labels that identify the versions is only that,
  > >    apparent.  But there is a disadvantage to such version-dependent
  > >    labels: when an older application receives data accompanied by a
  > >    newer, unknown label, it may fail to recognize the label and be
  > >    completely unable to deal with the data, whereas a generic, known
  > >    label would have triggered mostly correct processing of the data,
  > >    which may well not contain any new characters.
  > >
  > >    ["Korean mess" paragraph elided]
  > >
  > >    In practice, then, a version-independent label is warranted,
provided
  > >    the label is understood to refer to all versions after Amendment 5,
  > >    and provided no incompatible change actually occurs.  Should
  > >    incompatible changes occur in a later version of ISO/IEC 10646, the
  > >    MIME charset label defined here will stay aligned with the previous
  > >    version until and unless the IETF specifically decides otherwise.
  > >
  > >
  > >I think the same argument can apply to any other charset that evolves
  > >compatibly, i.e. later versions are strict supersets of earlier ones
and
  > >nothing else creates incompatibility.  Jonathan tells us that this is
the
  > >case with ISO_8859-8:1988 and :1999.
  > >
  > >This does not prevent the IANA registry from containing superset
  > >relationship information.
  > >
  > >Regards,
  > >
  > >--
  > >Fran輟is Yergeau
  >
  >