[Bug 12897] New: UTF-8 BOM should trump users and/or HTTP (Encoding sniffing algorithm)

http://www.w3.org/Bugs/Public/show_bug.cgi?id=12897

           Summary: UTF-8 BOM should trump users and/or HTTP (Encoding
                    sniffing algorithm)
           Product: HTML WG
           Version: unspecified
          Platform: PC
               URL: http://dev.w3.org/html5/spec/parsing#encoding-sniffing
                    -algorithm
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P3
         Component: HTML5 spec (editor: Ian Hickson)
        AssignedTo: ian@hixie.ch
        ReportedBy: xn--mlform-iua@xn--mlform-iua.no
         QAContact: public-html-bugzilla@w3.org
                CC: mike@w3.org, public-html-wg-issue-tracking@w3.org,
                    public-html@w3.org


PROPOSAL: 
   Spec IE and Webkit's handling of the Byte Order Mark for the UTF-8 encoding
as  REQUIRED:   Whenever the document begins with the UTF-8 Byte Order Mark,
then ignore the encoding info of the HTTP "Content-Type: text/html;
charset=[encodingname]" header and ignore as well any user actions to override
the document's encoding.
   Consequently, 
      * when there is a UTF-8 BOM,  then the encoding info provided by HTTP and
the user should be treated as irrelevant
      * the two first steps of the encoding sniffing algorithm must be changed 

CURRENT STATUS: 
   The encoding sniffing algorithm two first steps give users + transporation
layer (HTTP/MIME) power to override a document's character encoding:

   ]] 1. If the user has explicitly instructed the user agent to override the
document's character encoding with a specific encoding, optionally return that
encoding with the confidence certain and abort these steps.
      2. If the transport layer specifies an encoding, and it is supported,
return that encoding with the confidence certain, and abort these steps.[[

   HOWEVER, reality is that two mayor user agents operates with an exceptio to
the above rules:  Whenever the document includes the UTF-8 Byte Order Mark,
then Internet Explorer and Webkit  
    - do *not* allow users to override the encoding
    - do *not* respect the encoding information in the HTTP server's
Content-Type header.
    - do *not* permit their heuristic character dection features to guess any
encoding other than UTF-8
   Consequently, in IE and Webkit it is impossible for the user - as well as
for a HTTP server -  to cause a document with the UTF-8 Byte Order Mark to be
intepreted as e.g. KOI8-R encoded or Windows-1252 encoded.

   In contrast, Firefox and Opera
    - *do* obey the HTTP server's Content-Type header also when ther is a UTF-8
BOM 
    - *do* allow users to override the encoding also when ther is a UTF-8 BOM
    - *do* permit their heuristic character dection features to guess an
encoding other than UTF-8 (Opera and Firefox allow their users to tune/fiddle
with how their heuristic encoding sniffing work.)
   Consequently, in Firefox and Opera it is *possible* for the user - as well
as for a HTTP server -  to cause a document with the UTF-8 Byte Order Mark to
be intepreted as e.g. KOI8-R encoded or Windows-1252 encoded

BENEFITS:
    A. Harmonization with XML 1.0  Appendix F.2, "Priorities in the Presence of
External Encoding Information", which recommends BOM to have higher priority
than external encoding information:
http://www.w3.org/TR/xml/#sec-guessing-with-ext-info  (Opera/Firefox do not yet
implement this XML 1.0 recommendation)
    B. a simple, reliable way to specify the UTF-8 encoding
    C. FIrefox/Opera converge with IE/Webkit = browsers more interopable: 
    D. security: cameleon documens (where the document gets another and risky
interpretation when read as legacy encoding) become more difficult to create
    E. User experience: less "gibberish" and less "mojibake" for users
[*] http://en.wikipedia.org/wiki/Mojibake
    F. Same as A): Promotes a polyglot way to specify the encoding: the BOM
works in both HTML and XML. (The Polyglot spec already says that the UTF-8 BOM 
is the most polyglot enocoding method.)

Other justifications:
   - Opera Software "We have introduced the BOM as requirement for each source
file when we have written the build tools as a simple way to verify that all
files are utf8 encoded”.
(http://stackoverflow.com/questions/4658985/how-to-keep-the-bom-when-editing-files-in-espresso)

NOTES: 

   (1) Browsers tested as part of this bug report: IE8, Safari, Chrome (which
shows above described behavior) as well as Opera and Firefox (which do support
this behavior). Other browsers, e.g. KHTML, have not been tested.
   (2) BOM in UTF-16: I have not looked into how BOM in UTF-16 is handled by
parsers.
   (3)  For the record: All browsers, including Firefox and Opera, *do* already
ignore the META charset *element* whenever ther is a UTF-8 BOM. This bug report
says that they should *also* ignore HTTP.

-- 
Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

Received on Monday, 6 June 2011 20:02:01 UTC