I18N-ISSUE-455 (BUG28141): treatment of invalid 2-byte sequence is different in different CJK encodings [encoding]

I18N-ISSUE-455 (BUG28141): treatment of invalid 2-byte sequence is different  in different CJK encodings [encoding]

http://www.w3.org/International/track/issues/455

Raised by: Richard Ishida
On product: encoding

https://www.w3.org/Bugs/Public/show_bug.cgi?id=28141

This issue tracks the bug listed above and was created as part of the WG CR process.

---

Reporter: jshin@chromium.org

Per bug 16691 comment 15, I'm tightening Blink's encoding tables for CJK
encodings to handle unmappable 2-byte sequence in a safe manner. 



The current spec has the following provision after looking up |pointer|. 

* EUC-KR decoder
   If pointer is null and byte is in the range 0x00 to 0x7F, prepend byte to
stream.


* Big5 decoder

   If pointer is null and byte is in the range 0x00 to 0x7F, prepend byte to
stream.

* Shift_JIS decoder
   If pointer is null, prepend byte to stream.

* EUC-JP decoder
   If byte is not in the range 0xA1 to 0xFE, prepend byte to stream.


* GB18030 decoder
   If pointer is null, prepend byte to stream.

For now, let's put aside EUC-JP and GB18030. 

I don't see a reason to make SJIS decoder behave differently than EUC-KR and
Big5 decoder. One possible reason may be that [xA1, xDF] is a character by
itself. 

In SJIS, "0xFC 0xE0" [1] is turned to U+FFFD, but the second byte (0xE0)
becomes the lead of what follows.

In EUC-KR, "0xFE 0xE0" is turned to U+FFFD and the next lead byte is taken from
the byte *after* 0xE0. 

Shouldn't we change the part of SJIS decoder quoted above to the following? 

  If pointer is null and byte is in the range of 0x00 - 0x7F, prepend byte to
the stream.

Received on Monday, 30 March 2015 14:44:31 UTC