W3C home > Mailing lists > Public > public-i18n-core@w3.org > January to March 2008

Re: several messages about handling encodings in HTML

From: Mark Davis <mark.davis@icu-project.org>
Date: Thu, 28 Feb 2008 17:42:58 -0800
Message-ID: <30b660a20802281742v4c920110kac53a1ab0dc6d75a@mail.gmail.com>
To: "Ian Hickson" <ian@hixie.ch>
Cc: public-i18n-core@w3.org
On Thu, Feb 28, 2008 at 5:21 PM, Ian Hickson <ian@hixie.ch> wrote:
>
>  Executive summary: I made a number of changes, as described below, in
>  response to the feedback on character encodings in HTML. They are covered
>  by revisions 1263 to 1275 of the spec source.
>
>  I have cc'ed most (though not all) of the mailing lists that were
>  originally cc'ed on the messages to which I reply below, to keep everyone
>  in the loop. Please, for everyone's sake, pick a single mailing list when
>  replying, and trim the quotes to just the bits to which you are replying.
>  Don't include the whole of this e-mail in your reply! Thanks.
>
>
>  On Sun, 5 Nov 2006, Øistein E. Andersen wrote, in reply to Henri:
>  > >
>  > > I think conforming text/html documents should not be allowed to parse
>  > > into a DOM that contains characters that are not allowed in XML 1.0.
>  > > [...] I am inclined to prefer [...] U+FFFD
>
>  (I've made the characters not allowed in XML also not allowed in HTML,
>  with the exception of some of the space characters which we need to have
>  allowed for legacy reasons.)
>
>
>  > I perfectly agree. (Actually, i think that U+7F (delete) and the C1
>  > control characters should be excluded [transformed into U+FFFD] as well,
>  > but this could perhaps be problematic due to spurious CP1252
>  > characters.)
>
>  I've made them illegal but not converted them to FFFD.
>
>
>  On Mon, 6 Nov 2006, Lachlan Hunt wrote:
>  >
>  > At the very least, ISO-8859-1 must be treated as Windows-1252.  I'm not
>  > sure about the other ISO-8859 encodings.  Numeric and hex character
>  > references from 128 to 159 must also be treated as Windows-1252 code
>  > points.
>
>  All already specified.
>
>
>  On Sun, 5 Nov 2006, Elliotte Harold wrote:
>  >
>  > The specific problem is that an author may publish a correctly labeled
>  > UTF-8 or ISO-8859-8 document or some such. However the server sends a
>  > Content-type header that requires the parser to treat the document as
>  > ISO-8859-1 or US-ASCII or something else.
>  >
>  > The need is for server administrators to allow content authors to
>  > specify content types and character sets for the documents they write.
>  > The content doesn't need to change. The authors just need the ability to
>  > specify the server headers for their documents.
>
>  Well, we can't change the way this works from this side, so it's not
>  really our problem at this point.
>
>
>  On Sat, 23 Dec 2006, Henri Sivonen wrote:
>  >
>  > http://www.elementary-group-standards.com/web-standards/html5-http-equiv-difference.html
>  >
>  > In short, some authors want to use <meta http-equiv="imagetoolbar"
>  > content="no"> but (X)HTML5 doesn't allow it.
>  >
>  > Personally, I think that authors who want to disable *User* Agent
>  > features like that are misguided.
>  >
>  > Anyway, I thought I'd mention this so that the issue gets informed as
>  > opposed to accidental treatment.
>
>  Proprietary extensions to HTML are just that, proprietary extensions, and
>  are therefore by intentionally not conforming.
>
>
>  On Mon, 26 Feb 2007, Lachlan Hunt wrote:
>  >
>  > Given that the spec now says that ISO-8859-1 must be treated as
>  > Windows-1252, should it still be considered an error to use the C1
>  > control characters (U+0080 to U+009F) if ISO-8859-1 is declared?
>  >
>  > Some relevant messages from IRC:
>  >
>  > [15:59]       <Lachy> since the spec says if ISO-8859-1 is declared, Windows-1252
>  > must be used. Is it still an error for authors to use the C1 control
>  > characters in the range 128-159?
>  > [16:23]       <Hixie> Lachy: not sure what we should do, there's a bunch of corner
>  > cases there. like, should we allow control chars anyway, should we allow
>  > ISO-8859-1 to be declared but Win1252 to be used, etc.
>  > [16:23]       <Hixie> Lachy: can you mail the list with suggestions and a list of
>  > the cases you can think of that we should cover?
>  > [16:27]       <Lachy> I'm having a hard time deciding if it should be allowed or not
>  > [16:28]       <Lachy> Technically, it is an error and I think users should be
>  > notified, but it's practically harmless these days and very common.
>  > [16:30]       <Lachy> Yet, doing the same thing in XML doesn't work, since XML
>  > parsers do treat them as control characters
>
>  I've made it be a parse error. I'm sure implementing this is going to very
>  exciting for Henri.
>
>
>  On Thu, 1 Mar 2007, Henri Sivonen wrote:
>  >
>  > I think that encoding information should be included in the HTTP
>  > payload. In my opinion, the spec should not advice against this.
>  > Preferably, it would encourage putting the encoding information in the
>  > payload. (The BOM or, in the case of XML, the UTF-8 defaulting of the
>  > XML sniffing algorithm are fine.)
>
>  I can't seem to find the part of the spec that recommends the opposite of
>  this... did I already remove it? I'm happy to make the spec silent on this
>  point, since experts disagree.
>
>
>  On Sun, 11 Mar 2007, Geoffrey Sneddon wrote:
>  >
>  > From implementing parts of the input stream (section 8.2.2 as of
>  > writing) yesterday, I found several issues (some of which will show the
>  > asshole[1] within me):
>  >
>  >       - Within the step one of the get an attribute sub-algorithm it says
>  > "start over" – is this start over the sub-algorithm or the whole algorithm?
>
>  Fixed.
>
>
>  >       - Again in step one, why do we need to skip whitespace in both the
>  > sub-algorithm and at section one of the inner step for <meta> tags?
>
>  Otherwise, the <meta bit would be pointing at the "<" and would treat
>  "meta" as an attribute name.
>
>
>  >       - In step 11, when we have anything apart from a double/single quote
>  > or less/greater than sign, we add it to the value, but don't move the position
>  > forward, so when we move onto step 12 we add it again.
>
>  Yes, valid point. Fixed.
>
>
>  >       - In step 3 of the very inner set of steps for a content attribute in
>  > a <meta> tag, is charset case-sensitive?
>
>  Doesn't matter, the parser lowercases everything anyway.
>
>
>  >       - Again there, shouldn't we be given unicode codepoints for that (as
>  > it'll be a unicode string)?
>
>  Not sure what you mean.
>
>
>  On Sat, 26 May 2007, Henri Sivonen wrote:
>  >
>  > The draft says:
>  > "A leading U+FEFF BYTE ORDER MARK (BOM) must be dropped if present."
>  >
>  > That's reasonable for UTF-8 when the encoding has been established by
>  > other means.
>  >
>  > However, when the encoding is UTF-16LE or UTF-16BE (i.e. supposed to be
>  > signatureless), do we really want to drop the BOM silently? Shouldn't it
>  > count as a character that is in error?
>
>  Do the UTF-16LE and UTF-16BE specs make a leading BOM an error?
>
>  If yes, then we don't have to say anything, it's already an error.
>
>  If not, what's the advantage of complaining about the BOM in this case?
>
>
>  > Likewise, if an encoding signature BOM has been discarded and the first
>  > logical character of the stream is another BOM, shouldn't that also
>  > count as a character that is in error?
>  >
>  > I think I should elaborate that when the encoding is UTF-16 (not
>  > UTF-16LE or UTF-16BE), the BOM is gets swallowed by the character
>  > decoding layer (in reasonable decoder implementations) and is not
>  > returned from the character stream at all. Therefore, on the character
>  > level, a droppable BOM only occurs in UTF-8 when the encoding was
>  > established by other means.
>
>  The spec says: "Given an encoding, the bytes in the input stream must be
>  converted to Unicode characters for the tokeniser, as described by the
>  rules for that encoding, except that leading U+FEFF BYTE ORDER MARK
>  characters must not be stripped by the encoding layer."
>
>
>  On Mon, 28 May 2007, Henri Sivonen wrote:
>  >
>  > To this end, I think at least for conforming documents the algorithm for
>  > establishing the character encoding should be deterministic. I'd like to
>  > request two things:
>  >
>  > 1) When sniffing for meta charset, the current draft allows a use agent
>  > to give up sooner than after examining the first 512 bytes. To make meta
>  > charset sniffing reliable and deterministic so that it doesn't depend on
>  > flukes in buffering, I think UAs should (if there's no transfer protocol
>  > level charset label and no BOM) be required to consumer bytes until they
>  > find a meta charset, reach the EOF or have examined 512 bytes. That is,
>  > I think UAs should not be allowed to give up earlier. (On the other
>  > hand, I think UAs should be allowed to start examining the byte stream
>  > before 512 have been buffered without an IO error, since in general,
>  > byte stream buffer management should be up to the IO libraries and
>  > outside the scope of the HTML spec.)
>
>  I don't want to do this because I don't want to require that browsers
>  handle a CGI script that outputs 500 bytes than hangs for a minute in a
>  way that doesn't render anything for a minute, and I don't want to require
>  that people writing such CGI scripts front-load a 512 byte comment.
>
>  We've already conceeded that a page can document.write() an encoding
>  declaration after 6 megabytes of content and end up causing a reparse.
>
>
>  > 2) Since the chardet step is optional and the spec doesn't make the
>  > Mozilla chardet behavior normative, I think the document should be
>  > considered non-conforming if the algorithm for establishing the
>  > character encoding proceeds to steps 6 (chardet) or 7 (last resort
>  > default).
>
>  That would make most of my pages non-conforming. It would make this
>  non-conforming:
>
>    <!DOCTYPE HTML>
>    <html>
>     <head>
>      <title> Example </title>
>     </head>
>     <body>
>      <p> I don't want to be non-conforming! </p>
>     </body>
>    </html>
>
>
>  > It wouldn't hurt, though, to say in the section on writing documents that at
>  > least one of the following is required for document conformance:
>  >  * A transfer protocol-level character encoding declaration.
>  >  * A meta charset within the first 512 bytes.
>  >  * A BOM.
>
>  We already require that, though without the 512 byte requirement.
>
>
>  On Tue, 29 May 2007, Henri Sivonen wrote:
>  >
>  > To avoid stepping on the toes of Charmod more than is necessary, I
>  > suggest making it non-conforming for a document to have bytes in the
>  > 0x80…0x9F range when the character encoding is declared to be one of the
>  > ISO-8859 family encodings.
>
>  Done, I believe.
>
>
>  > (UA conformance requires in some cases these bytes to be decoded in a
>  > Charmod-violating way, but reality trumps Charmod for UA conformance.
>  > While I'm at it: Surely there are other ISO-8859 family encodings
>  > besides ISO-8859-1 that require decoding using the corresponding
>  > windows-* family decoder?)
>
>  Maybe; anyone have any concrete information?
>
>
>  On Tue, 29 May 2007, Maciej Stachowiak wrote:
>  >
>  > I don't know of any ISO-8859 encodings requiring this, but for all
>  > unicode encodings and numeric entity references compatibility requires
>  > interpreting this range of code points in the WinLatin1 way.
>
>  On Mon, 4 Jun 2007, Henri Sivonen wrote:
>  >
>  > I tested with Firefox 2.0.4, Minefield, Safari 2.0.4, WebKit nightly and
>  > Opera 9.20 (all on Mac). Only Safari 2.0.4 gives the DWIM treatment the
>  > C1 code point range in UTF-8 and UTF-16.
>  >
>  > This makes me suspect that compatibility with the Web doesn't really
>  > require the DWIM treatment here. What does IE7 do?
>  >
>  > The data I used: http://hsivonen.iki.fi/test/utf-c1/
>
>  IE7 and Safari 3 do the same as the other browsers, namely, no DWIM
>  treatment.
>
>  So, I haven't changed the spec.
>
>
>  On Fri, 1 Jun 2007, Henri Sivonen wrote:
>  >
>  > The anomalies seem to be:
>  >  1) ISO-8859-1 is decoded as Windows-1252.
>  >  2) 0x85 in ISO-8859-10 and in ISO-8859-16 is decoded as in Windows-1252
>  > (ellipsis) by Gecko.
>  >  3) ISO-8859-11 is decoded as Windows-874.
>  >
>  > I was rather surprised by the results. They weren't at all what I expected.
>  > Test data: http://hsivonen.iki.fi/test/iso8859/
>  >
>  > I suggest adding the ISO-8859-11 to Windows-874 mapping to the spec.
>
>  On Fri, 1 Jun 2007, Henri Sivonen wrote:
>  >
>  > By Firefox and Opera. Safari doesn't support ISO-8859-11 and I was
>  > unable to test IE.
>
>  On Fri, 1 Jun 2007, Simon Pieters wrote:
>  >
>  > IE7 and Opera handle ISO-8859-11.htm the same, AFAICT.
>
>  I did some studies and there appear to be enough pages as ISO-8859-11 to
>  add this. I didn't check how many had bytes in the affected range, which
>  maybe would be worth checking, though.
>
>
>  On Sat, 2 Jun 2007, Øistein E. Andersen wrote:
>  >
>  > As suggested earlier [1], a simpler solution seems to be to treat C1
>  > bytes and NCRs from /all/ ISO-8859-* and Unicode encodings as
>  > Windows-1252.
>
>  That seems excessive.
>
>
>  On Tue, 5 Jun 2007, Henri Sivonen wrote:
>  > >
>  > > To avoid stepping on the toes of Charmod more than is necessary, I
>  > > suggest making it non-conforming for a document to have bytes in the
>  > > 0x80…0x9F range when the character encoding is declared to be one of
>  > > the ISO-8859 family encodings.
>  >
>  > I've been thinking about this. I have a proposal on how to spec this
>  > *conceptually* and how to implement this with error reporting. I am
>  > assuming here that 1) No one ever intends C1 code points to be present
>  > in the decoded stream and 2) we want, as a Charmod correctness fig leaf,
>  > to make the C1 bytes non-conforming when ISO-8859-1 or ISO-8859-11 was
>  > declared but Windows-1252 or Windows-874 decoding is needed.
>
>  I really don't care too much about the fig leaf part.
>
>
>  > Based on the behavior of Minefield and Opera 9.20, the following seems
>  > to be the least Charmod violating and least quirky approach that could
>  > possibly work:
>  >
>  > 1) Decode the byte stream using a decoder for whatever encoding was declared,
>  > even ISO-8859-1 or ISO-8859-11, according to ftp://
>  > ftp.unicode.org/Public/MAPPINGS/.
>  > 2) If a character in the decoded character stream is in the C1 code point
>  > range, this is a document conformance violation.
>  >    2a) If the declared encoding was ISO-8859-1, replace that character with
>  > the character that you get by casting the code point into a byte and decoding
>  > it as Windows-1252.
>  >    2b) If the declared encoding was ISO-8859-11, replace that character with
>  > the character that you get by casting the code point into a byte and decoding
>  > it as Windows-874.
>
>  That sounds far more complex than what we have now.
>
>
>  On Tue, 5 Jun 2007, Kristof Zelechovski wrote:
>  >
>  >     2c) If the declared encoding was ISO-8859-2, replace that character
>  > with the character that you get by casting the code point into a byte
>  > and decoding it as Windows-1250.
>
>  On Tue, 5 Jun 2007, Henri Sivonen wrote:
>  >
>  > As far as I can tell, that's not what Firefox, Minefield, Opera 9.20 and
>  > WebKit nightlies do, so apparently it is not required for compatibility
>  > with a notable number of pages.
>
>  Indeed.
>
>
>  On Tue, 5 Jun 2007, Maciej Stachowiak wrote:
>  >
>  > What we actually do in WebKit is always use a windows-1252 decoder when
>  > ISO-8859-1 is requested. I don't think it's very helpful to make all
>  > documents that declare a ISO-8859-1 encoding and use characters in the
>  > C1 range nonconforming. It's true that they are counting on nonstandard
>  > processing of the nominally declared encoding, but I don't think that
>  > causes a problem in practice, as long as the rule is well known. It
>  > seems simpler to just make latin1 an alias for winlatin1.
>
>  I agree.
>
>
>  On Fri, 1 Jun 2007, Raphael Champeimont (Almacha) wrote:
>  >
>  > I think there is something wrong in the "get an attribute" algorithm
>  > from 8.2.2. The input stream.
>  >
>  > Between steps 11 and 12 I think there is a missing:
>  >
>  > 11b: Advance position to the next byte.
>  >
>  > With the current algorithm, if I write <meta charset = ascii> it will
>  > say the value of attribute charset is "aascii" with one too much leading
>  > A
>  >
>  > The reason is that in step 11 if we fall in case "Anything else" we add
>  > the new char to the string, and then if we fall in "Anything else" in
>  > step 12 we add again the *same* char to the string, so the first char of
>  > the attribute value appears 2 times.
>
>  Fixed. (Though please check. I made several changes to this algorithm and
>  would be happier if I knew someone had proofread the changes!)
>
>
>  On Fri, 1 Jun 2007, Henri Sivonen wrote:
>  >
>  > In the charset meta sniffing algorithm under "Attribute name:":
>  >
>  > > If it is 0x2F (ASCII '/'), 0x3C (ASCII '<'), or 0x3E (ASCII '>')
>  > >     Stop looking for an attribute. The attribute's name is the value of
>  > > attribute name, its value is the empty string.
>  >
>  > In general, it seems to me the algorithm isn't quite clear on when to
>  > stop looking for the current attribute and when to stop looking for
>  > attributes for the current tag altogether.
>
>  The spec never distinguishes these two cases in the "get an attribute"
>  algorithm -- the algorithm that invokes the "get an attribute" algorithm
>  is the one that decides how often it is done.
>
>
>  > In this step, it seems to me that '/' should advance the pointer and end
>  > getting the current attribute followed by getting another attribute. '>'
>  > should end getting attributes on the whole tag without changing the
>  > pointer.
>
>  It doesn't matter. Both return an attribute, then the invoking algorithm
>  retries and if that results in no attribute (because you're on the ">")
>  then you stop looking for the tag.
>
>
>  On Fri, 1 Jun 2007, Henri Sivonen wrote:
>  >
>  > The spec probably needs to be made more specific about the case where
>  > the ASCII byte-based algorithm finds a supported encoding name but the
>  > encoding is not a rough ASCII superset.
>  >
>  > 23:46 < othermaciej> one quirk in Safari is that if there's a meta tag
>  > claiming
>  >                      the source is utf-16, we treat it as utf-8
>  > ...
>  > 23:48 < othermaciej> hsivonen: there is content that needs it
>  > ...
>  > 23:52 < othermaciej> hsivonen: I think we may treat any claimed unicode
>  > charset
>  >                      in a <meta> tag as utf-8
>
>  Oops, I had this for the case where utf-16 was detected on the fly, but
>  not for the preparser. Fixed.
>
>
>  On Sat, 2 Jun 2007, Philip Taylor wrote:
>  >
>  > 8.2.2. The input stream: "If the next six characters are not 'charset'"
>  > - s/six/seven/
>
>  Fixed.
>
>
>  On Thu, 14 Jun 2007, Henri Sivonen wrote:
>  >
>  > As written, the charset sniffing algorithm doesn't trim space characters
>  > from around the tentative encoding name. html5lib test case expect the
>  > space characters to be trimmed.
>  >
>  > I suggest trimming space characters (or anything <= 0x20 depending on
>  > which approach is the right for compat).
>
>  Actually it seems browsers don't do any trimming here. I've added a
>  comment to that effect.
>
>
>  On Sat, 23 Jun 2007, Øistein E. Andersen wrote:
>  > >>
>  > >>> Bytes or sequences of bytes in the original byte stream that could
>  > >>> not be converted to Unicode characters must be converted to U+FFFD
>  > >>> REPLACEMENT CHARACTER code points.
>  > >>
>  > >> [This does not specify the exact number of replacement chracters.]
>  > >
>  > > I don't really know how to define this.
>  >
>  > Unicode 5.0 remains vague on this point. (E.g., definition D92 defines
>  > well-formed and ill-formed UTF-8 byte sequences, but conformance
>  > requirement C10 only requires ill-formed sequences to be treated as an
>  > error condition and suggests that a one-byte ill-formed sequence may be
>  > either filtered out or replaced by a U+FFFD replacement character.) More
>  > generally, character encoding specifications can hardly be expected to
>  > define proper error handling, since they are usually not terribly
>  > preoccupied with mislabelled data.
>
>  They should define error handling, and are defective if they don't.
>  However, I agree that many specs are defective. This is certainly not
>  limited to character encoding specifications.
>
>
>  > The current text may nevertheless be two liberal. It would notably be
>  > possible to construct an arbitrarily long Chinese text in a legacy
>  > encoding which -- according to the spec -- could be replaced by one
>  > single U+FFFD replacement character if incorrectly handled as UTF-8,
>  > which might lead the user to think that the page is completely
>  > uninteresting and therefore move on, whereas a larger number of
>  > replacement characters would have led him to try another encoding. (This
>  > is only a problem, of course, if an implementor chooses to emit the
>  > minimal number of replacement characters sanctioned by the spec.)
>
>  Yes, but this is a user interface issue, not an interoperability issue, so
>  I don't think we need to be concerned about it.
>
>
>  On Thu, 2 Aug 2007, Henri Sivonen wrote:
>
>  > On Aug 2, 2007, at 10:11, Ian Hickson wrote:
>  >
>  > > Would a non-normative note help here? Something like:
>  > >
>  > >    Note: Bytes or sequences of bytes in the original byte stream that did
>  > >    not conform to the encoding specification (e.g. invalid UTF-8 byte
>  > >    sequences in a UTF-8 input stream) are errors that conformance
>  > >    checkers are expected to report.
>  > >
>  > > ...to be put after the paragraph that reads "Bytes or sequences of
>  > > bytes in the original byte stream that could not be converted to
>  > > Unicode characters must be converted to U+FFFD REPLACEMENT CHARACTER
>  > > code points".
>  >
>  > Yes, this is what I meant with "a note hinting the consequences.
>
>  Ok, added.
>
>
>  > > (Note that not all bytes or sequences of bytes in the original byte
>  > > stream that could not be converted to Unicode characters are
>  > > necessarily errors. It could just be that the encoding has a character
>  > > set that isn't a subset of Unicode, e.g. the Apple logo found in most
>  > > Apple character sets doesn't have a non-PUA analogue in Unicode. Its
>  > > presence in an HTML document isn't an error as far as I'm concerned.)
>  >
>  > Since XML and HTML5 are defined in terms of Unicode, characters there's
>  > nowhere to go except error and REPLACEMENT CHARACTER or the PUA for
>  > characters that aren't in Unicode. I'd steer clear of this in the spec
>  > an let decoders choose between de facto PUA assignments (like U+F8FF for
>  > the Apple logo) and errors.
>
>  Yeah I don't have any intention on mentioning this in the spec.
>
>
>  On Wed, 31 Oct 2007, Martin Duerst wrote:
>  >
>  > [8.2.2.1]
>  >
>  > In point 3., it's not completely clear whether the encoding returned is
>  > e.g. "UTF-16BE BOM" or "UTF-16BE". Probably the best thing editorially
>  > is to move the word BOM from the description column of the table to the
>  > text prior to the table.
>
>  Fixed.
>
>
>  > In point 7, what I find unnecessary is the repeated mention of heuristic
>  > algorithms, which are already mentioned previously in point 6.
>
>  The heuristics in step 6 are for detemrining an encoding based on the byte
>  stream, e.g. using frequency analysis. The heuristics in step 7 are for
>  picking a default once that has failed. For example, if the defaults are
>  UTF-8 or Win1252, then you can determine which to pick by simply deciding
>  whether or not the stream is valid UTF-8.
>
>
>  > (I'm really interested what document [UNIVCHADET] is going to point to.)
>
>  http://www.mozilla.org/projects/intl/UniversalCharsetDetection.html
>
>  (It's in the source.)
>
>
>  > What I find missing/unclear is that the user can overwrite the page
>  > encoding manually. What is mentioned is a user-specificed default, which
>  > makes sense (e.g. "well, I'm mostly viewing Chinese pages, so I set my
>  > default to GB2132"). However, what we also need is the possibility for a
>  > user to override the encoding of a specific page (not changing the
>  > default). This is necessary because some pages are still mislabeled.
>  > When such an override is present, it should come before what's currently
>  > number 1.
>
>  User agents can provide user interfaces to override anything they want,
>  e.g. they could provide an interface that changes all <script> elements
>  into <pre> elements on the fly, or whatever. Such behaviour is outside the
>  scope of the specification, since it is no longer about interoperability,
>  but about user control. It's technically non-compliant, because it is
>  doing something with the page that doesn't match what would happen for
>  other people (unless they _also_ overrode the spec behaviour).
>
>
>  > In 8.2.2.2, what I find unnecessary is that encodings such as UTF-7 are
>  > explicitly forbidden. I agree that these are virtually useless. However,
>  > I don't think implementing them would create any harm, and I don't think
>  > they should be dignified by even mentioning them.
>
>  Sadly they do cause harm. The ones that are outlawed have all been used in
>  eithir actual attacks or proof-of-concept attacks described in
>  vulnerability reports, mostly due to their deceptive similarity to more
>  common encodings. (UTF-7 in particular has been used in a number of
>  attacks, because IE supported auto-detecting it, if I recall correctly.)
>
>
>  > In 8.2.2.4, I have no idea what's the reason or purpose of point 1,
>  > which reads "If the new encoding is UTF-16, change it to UTF-8.". I
>  > suspect some misunderstanding.
>
>  This is required because many pages are labelled as UTF-16 but actually
>  use UTF-8. For example:
>
>   http://www.zingermans.com
>
>
>  > Well, now let's get back to CharMod, and to the place where I think you
>  > need to do more work. HTML5 currently says "treat data labeled
>  > iso-8859-1 as windows-1252". This conflicts with C025 of CharMod
>  > (http://www.w3.org/TR/charmod/#C025):
>  >
>  > C025 [I] [C] An IANA-registered charset name MUST NOT be used to label
>  > text data in a character encoding other than the one identified in the
>  > IANA registration of that name.
>  >
>  > and also C030 (http://www.w3.org/TR/charmod/#C030): C030 [I] When an
>  > IANA-registered charset name is recognized, receiving software MUST
>  > interpret the received data according to the encoding associated with
>  > the name in the IANA registry.
>  >
>  > So the following sentence:
>  >
>  > "When a user agent would otherwise use the ISO-8859-1 encoding, it must
>  > instead use the Windows-1252 encoding."
>  >
>  > from HTML5 is clearly not conforming to CharMod.
>
>  Indeed, it says so explicitly in the spec.
>
>
>  > Please note that the above items (C025 and C030) say that they only
>  > affect implementations ([I]) and content ([C]), but I think the main
>  > reason for this is that we never even immagined that a spec would say
>  > "you must treat FOO as BAR".
>  >
>  > I don't disagree with 'widely deployed', but I think one main reason for
>  > this is that it took ages to get windows-1252 registered. I think there
>  > are other ways to deal with this issue than a MUST. One thing that I
>  > guess you could do is to just describe current practice.
>
>  Well, what we're describing is what an implementation has to do to be
>  compatible with the other implementations. And right now, this is one of
>  the things it has to do.
>
>
>  > This brings me to another point: The whole HTML5 spec seems to be
>  > written with implementers, and implementers only, in mind. This is great
>  > to help get browser behavior aligned, but it creates an enormous
>  > problem: The majority of potential users of the spec, namely creators of
>  > content, and of tools creating content, are completely left out. As an
>  > example, trying to reverse-engineer how to indicate the character
>  > encoding inside an HTML5 document from point 4 in 8.2.2.1 is completely
>  > impossible for content creators, webmasters, and the like.
>
>  Section "8.2 Parsing HTML documents" is indeed exclusively for user agent
>  implementors and conformance checker implementors. For authors and
>  authoring tool implementors, you want section "8.1 Writing HTML documents"
>  and section "3.7.5.4. Specifying the document's character encoding" (which
>  is linked to from 8.1). These give the flipside of these requirements, the
>  authoring side.
>
>
>  On Sat, 3 Nov 2007, Addison Phillips wrote:
>  >
>  > --
>  > Otherwise, return an implementation-defined or user-specified default
>  > character encoding, with the confidence tentative. Due to its use in
>  > legacy content, windows-1252 is recommended as a default in
>  > predominantly Western demographics. In non-legacy environments, the more
>  > comprehensive UTF-8 encoding is recommended instead. Since these
>  > encodings can in many cases be distinguished by inspection, a user agent
>  > may heuristically decide which to use as a default.
>  > --
>  >
>  > Our comment is that this is a pretty weak recommendation. It is
>  > difficult to say what a "Western demographic" means in this context. We
>  > think we know why this is here: untagged HTML4 documents have a default
>  > character encoding of ISO 8859-1, so it is unsurprising to assume its
>  > common superset encoding when no other encoding can be guessed.
>  >
>  > However, we would like to see several things happen here:
>  >
>  > 1. It never actually says anywhere why windows-1252 must be used instead
>  > of ISO 8859-1.
>
>  This is required in "Preprocessing the input stream".
>
>
>  > 2. As quoted, it seems to (but does not actually) favor 1252 over UTF-8.
>  > Since UTF-8 is highly detectable and also the best long-term general
>  > default, we'd prefer if the emphasis were reversed, dropping the
>  > reference to "Western demographics". For example:
>  >
>  > --
>  > Otherwise, return an implementation-defined or user-specified default
>  > character encoding, with the confidence tentative. UTF-8 is recommended
>  > as a default encoding in most cases. Due to its use in legacy content,
>  > windows-1252 is also recommended as a default. Since these encodings can
>  > usually be distinguished by inspection, a user agent may heuristically
>  > decide which to use as a default.
>  > --
>
>  I've reversed the order, though not removed the mention of the Western
>  demographic, which I think is actually quite accurate and genernally more
>  understandable than, say, occidental. I would like to know what the more
>  common codecs are in oriental demographics, though, to broaden the use of
>  the recommendations.
>
>
>  > 3. Possibly something should be said (elsewhere, not in this paragraph)
>  > about using other "superset" encodings in preference to the explicitly
>  > named encoding (that is, other encodings bear the same relationship as
>  > windows-1252 does to iso8859-1 and user-agents actually use these
>  > encodings to interpret pages and/or encode data in forms, etc.)
>
>  Is the current (new) text sufficient in this regard? See also the earlier
>  comments for details on the decisions behind the new text.
>
>
>  On Thu, 6 Dec 2007, Sam Ruby wrote:
>  > Ian Hickson wrote:
>  > > On Wed, 5 Dec 2007, Sam Ruby wrote:
>  > > > Henri Sivonen wrote:
>  > > > > I identified four classes of errors:
>  > > > >  1) meta charset in XHTML
>  > > > Why specifying a charset that matches the encoding is flagged as an
>  > > > error is probably something that should be discussed another day.
>  > > > I happen to believe that people will author content intended to be
>  > > > used by multiple user agents which are at various levels of spec
>  > > > conformance.
>  > >
>  > > That's actually an XML issue -- XML says the encoding should be in the
>  > > XML declaration, so HTML tries to not step on its toes and says that
>  > > the charset declaration shouldn't be included in the markup. (The spec
>  > > has to say that the UA must ignore that line anyway, so it's not clear
>  > > that there's any benefit to including it.)
>  >
>  > If the declaration clashed, I could see the value in an error message,
>  > but as I said, this can be discussed another day.
>
>  Is it another day yet? :-)
>
>
>  On Fri, 25 Jan 2008, Frank Ellermann wrote:
>  >
>  > Hi, the chapter about "acceptable" charsets (8.2.2.2) is messy. Clearly
>  > UTF-8 and windows-1252 are popular, and you have that.
>  >
>  > What you need as a "minimum" for new browsers is UTF-8, US-ASCII (as
>  > popular proper subset of UTF-8), ISO-8859-1 (as HTML legacy), and
>  > windows-1252 for the reasons stated in the draft, supporting Latin-1 but
>  > not windows-1252 would be stupid.
>
>  Right, that's what the draft current requires.
>
>
>  > BTW, I'm not aware that windows-1252 is a violation of CHARMOD, I asked
>  > a question about it and C049 in a Last Call of CHARMOD.
>
>  See one of the earlier e-mails in this compound reply for the reasoning.
>
>
>  > Please s/but may support more/but should support more/ - the minimum is
>  > only that, the minimum.
>
>  "SHOULD" has very strong connotations that I do not think apply here. In
>  particular, it makes no sense to have an open-ended SHOULD in this
>  context.
>
>
>  > | User agents must not support the CESU-8, UTF-7, BOCU-1 and SCSU
>  > | encodings
>  >
>  > I can see a MUST NOT for UTF-7 and CESU-8.  And IMO the only good excuse
>  > for legacy charsets is backwards compatibility.  But that is at worst a
>  > "SHOULD NOT" for BOCU-1, as you have it for UTF-32.
>  >
>  > I refuse to discuss SCSU, but MUST NOT is rather harsh, isn't it ?
>
>  As noted earlier, these requirements are derived from real or potential
>  security vulnerabilities.
>
>
>  > In 3.7.5.4 you say:
>  >
>  > | Authors should not use JIS_X0212-1990, x-JIS0208, and encodings
>  > | based on EBCDIC.  Authors should not use UTF-32.
>  >
>  > What's the logic behind these recommendations ?  Of course EBCDIC
>  > is rare (as far as HTML is concerned I've never seen it), but it's
>  > AFAIK not worse than codepage 437, 850, 858, or similar charsets.
>
>  Those are non-US-ASCII-compatible encodings. For further reasoning see the
>  thread that resulted in:
>
>    http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-June/011949.html
>
>
>  > And UTF-32 is relatively harmless, not much worse than UTF-16, it
>  > belongs to the charsets recommended in CHARMOD.  Depending on what
>  > happens in future Unicode versions banning UTF-32 could backfire.
>
>  Actually UTF-32 is quite harmful, due to its extra cost in implementation,
>  its very limited testing, and the resulting bugs in almost all known
>  implementations.
>
>
>  > There are lots of other charsets starting with UTF-1 that could be
>  > listed as SHOULD NOT or even MUST NOT.  Whatever you pick, state what
>  > your reasons are, not only the (apparently) arbitrary result.
>
>  The reasons are sometimes rather involved or subtle, and I'd rather not
>  have the specification defend itself. It's a spec, not a positon paper. :-)
>
>
>  > Please make sure that all *unregistered* charsets are SHOULD NOT. Yes, I
>  > know the consequences for some proprietary charsets, they are free to
>  > register them or to be ignored (CHARMOD C022).
>
>  It's already a must ("The value must be a valid character encoding name,
>  and must be the preferred name for that encoding.").
>
>
>  On Tue, 29 Jan 2008, Brian Smith wrote:
>  > Henri Sivonen wrote:
>  > > My understanding is that HTML 5 bans these post-UTF-8
>  > > second-system Unicode encodings no matter where you might
>  > > declare the use.
>  >
>  > It is in section 3.7.5 (the META element), and not in section 8 (The
>  > HTML Syntax), and the reference to section 3.7.5 in section 8 says that
>  > the restrictions apply (only) in a (<META>) character encoding
>  > declaration. So, it seems the real issue is just clarifying the text in
>  > 3.7.5.4 to indicate that those restrictions apply only when the META
>  > charset override mechanism is being used.
>
>  I don't understand.
>
>
>  > > The purpose of the HTML 5 spec is to improve interoperability between
>  > > Web browsers as used with content and Web apps published on the one
>  > > public Web. The normative language in the spec is concerned with
>  > > publishing and consuming content and apps on the Web. The purpose of
>  > > the spec isn't to lower the R&D cost of private and proprietary
>  > > systems by producing reusable bits.
>  >
>  > Then why doesn't the specification list the encodings that conformant
>  > web browsers are required to support, instead of listing the encodings
>  > that document authors are forbidden from using.
>
>  Because former the list is open-ended, whereas the latter list is not,
>  and the latter list is more important.
>
>
>  > > > Even after Unicode and the UTF encodings, new encodings are still
>  > > > being created.
>  > >
>  > > Deploying such encodings on the public network is a colossally bad
>  > > idea. (My own nation has engaged in this folly with ISO-8859-15, so
>  > > I've seen the bad consequences at home, too.)
>  >
>  > That is exactly my point. If the intention is that BOCU-1 should be
>  > prohibited, then shouldn't ISO-8859-15 be prohibited for the same
>  > reason? Why one and not the other?
>
>  One is used. The other is not. It really is that simple. We can stop the
>  madness for one of them, but it's too late for the other.
>
>
>  > Anyway, I am pretty sure that the restriction against BOCU and similar
>  > encodings is just to make it possible to correctly parse the <META>
>  > charset override, not to prevent their use altogether. The language just
>  > needs to be made clearer.
>
>  As the spec says, "authors must not use the CESU-8, UTF-7, BOCU-1 and SCSU
>  encodings". There's no limitation to <meta> or anything. They are just
>  banned outright.
>
>
>  On Thu, 31 Jan 2008, Henri Sivonen wrote:
>  >
>  > I ran an analysis on recent error messages from Validator.nu.
>  > http://hsivonen.iki.fi/test/moz/analysis.txt
>
>  Looking at this from the point of view of encodings, I see the following
>  common errors:
>
>   * <meta charset> not being at the top of <head>
>   * missing explicit character encoding declaration
>   * <meta content=""> not starting with text/html
>   * unpreferred encoding names
>
>  I think all of these are real errors, and I don't think we should change
>  the spec's encoding rules based on this data.
>
>  Thanks for this data. Basing spec development on real data like this is of
>  huge value.
>
>
>  On Thu, 31 Jan 2008, Sam Ruby wrote:
>  > >
>  > > I think we should allow the old internal encoding declaration syntax
>  > > for text/html as an alternative to the more elegant syntax. Not
>  > > declaring the encoding is bad, so we shouldn't send a negative message
>  > > to the authors who are declaring the encoding. Moreover, this is
>  > > interoperable stuff.
>  > >
>  > > I think we shouldn't allow this for application/xhtml+xml, though,
>  > > because authors might think it has an effect.
>  >
>  > By that reasoning, a meta charset encoding declaration should not be
>  > allowed if a charset is specified on the Content-Type HTTP header.  I
>  > ran into that very problem today:
>  >
>  > http://lists.planetplanet.org/archives/devel/2008-January/001747.html
>  >
>  > This content was XHTML, but was served as text/html, with a charset
>  > specified on the HTTP header, which overrode the charset on the meta
>  > declaration.
>
>  If they don't match, then there's an error (forcibly so, since one of the
>  two encodings has to be wrong!).
>
>
>  > Serving XHTML as text/html, with BOTH a charset specified on the HTTP
>  > header AND a meta charset specified just in case is more common than you
>  > might think.
>
>  It's not a recommended behaviour, though. Just pick one and use it. The
>  practice of making documents schizophrenic like this is a side-effect of
>  the market not fully supporting XHTML (i.e. IE). If it wasn't for that,
>  people wouldn't be as determined to give their documents identity crises.
>
>
>  > A much more useful restriction -- spanning both the HTML5 and XHTML5
>  > serializations -- would be to issue an error if multiple sources for
>  > encoding information were explicitly specified and if they differ.
>
>  That's already required.
>
>
>  On Mon, 11 Feb 2008, Henri Sivonen wrote:
>  > >
>  > > A much more useful restriction -- spanning both the HTML5 and XHTML5
>  > > serializations -- would be to issue an error if multiple sources for
>  > > encoding information were explicitly specified and if they differ.
>  >
>  > I agree. I had already implemented this as a warning on the XML side.
>  > (Not as an error because I'm not aware of any spec that I could justify
>  > for calling it an error.)
>
>  If the declarations disagree, one of them is wrong. It's an error for the
>  declaration to be wrong.
>
>
>  > While I was at it, I noticed that the spec (as well as Gecko) don't
>  > require http-equiv='content-type' when looking for a content attribute
>  > that looks like an internal encoding declaration. Therefore, I also
>  > added a warning that fires if the value of a content attribute would be
>  > sniffed as an internal character encoding declaration but a
>  > http-equiv='content-type' is missing.
>
>  It's an error according to the spec.
>
>
>  On Fri, 1 Feb 2008, Henri Sivonen wrote:
>  >
>  > But surely the value for content should be ASCII-case-insensitive.
>
>  Ok.
>
>
>  > Also, why limit the space to one U+0020 instead of zero or more space
>  > characters?
>
>  Ok, allowed any number of space characters (and any space characters).
>
>  --
>  Ian Hickson               U+1047E                )\._.,--....,'``.    fL
>  http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
>  Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'



-- 
Mark
Received on Friday, 29 February 2008 01:43:10 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 1 October 2008 10:18:53 GMT