Re: New full Unicode for ES6 idea

I'm not sure what to think about this, being a big fan of the UTF-8
simplicity. :) But anyhow, I like the idea of opt-in, actually so much that
I started thinking, why not make JS be encoding-agnostic?

What I mean here is that maybe we could have multi-charset Strings in JS?
This would be useful especially on the server-side JS. So, what I'm
suggesting is an extension to the String class, maybe defined as follows
(while being just the first thing from the top of my head).

Let's say we have a loadFile function that takes a filename and reads its
contents to a string it returns.

loadFile('my-utf8-file').charset === 'UTF-8'
String(loadFile('my-file'), 'UTF-16').charset === 'UTF-16'
loadFile('my-file').toString('UTF-9').charset === 'UTF-9'
32..toString(10, 'UTF-8').charset === 'UTF-8'

// And hence, we could add easy sugar to the function as well,
loadFile('my-utf8-file', 'UTF-16').charset === 'UTF-16';

What do you think?

Obviously this creates a lot of problems, but backwards compatibility could
(maybe) be preserved without an opt-in, if the default charset would stay
the same.

Cheers,
Jussi

On Sun, Feb 19, 2012 at 10:33 AM, Brendan Eich <brendan@mozilla.com> wrote:

> Once more unto the breach, dear friends!
>
> ES1 dates from when Unicode fit in 16 bits, and in those days, nickels had
> pictures of bumblebees on 'em ("Gimme five bees for a quarter", you'd say
> ;-).
>
> Clearly that was a while ago. These days, we would like full 21-bit
> Unicode character support in JS. Some (mranney at Voxer) contend that it is
> a requirement.
>
> Full 21-bit Unicode support means all of:
>
> * indexing by characters, not uint16 storage units;
> * counting length as one greater than the last index; and
> * supporting escapes with (up to) six hexadecimal digits.
>
> ES4 saw bold proposals including Lars Hansen's, to allow implementations
> to change string indexing and length incompatibly, and let Darwin sort it
> out. I recall that was when we agreed to support "\u{XXXXXX}" as an
> extension for spelling non-BMP characters.
>
> Allen's strawman from last year, http://wiki.ecmascript.org/**
> doku.php?id=strawman:support_**full_unicode_in_strings<http://wiki.ecmascript.org/doku.php?id=strawman:support_full_unicode_in_strings>,
> proposed a brute-force change to support full Unicode (albeit with too many
> hex digits allowed in "\u{...}"), observing that "There are very few places
> where the ECMAScript specification has actual dependencies upon the size of
> individual characters so the compatibility impact of supporting full
> Unicode is quite small." But two problems remained:
>
> P1. As Allen wrote, "There is a larger impact on actual implementations",
> and no implementors that I can recall were satisfied that the cost was
> acceptable. It might be, we just didn't know, and there are enough signs of
> high cost to create this concern.
>
> P2. The change is not backward compatible. In JS today, one read a string
> s from somewhere and hard-code, e.g., s.indexOf("0xd800" to find part of a
> surrogate pair, then advance to the next-indexed uint16 unit and read the
> other half, then combine to compute some result. Such usage would break.
>
> Example from Allen:
>
> var c = "😁" // where the single character between the quotes is the
> Unicode character U+1f638
>
> c.length == 2;
> c === "\ud83d\ude38"; //the two character UTF-16 encoding of 0x1f683
> c.charCodeAt(0) == 0xd83d;
> c.charCodeAt(1) == 0xd338;
>
> (Allen points out how browsers, node.js, and other environments blindly
> handle UTF-8 or whatever incoming format recoding to UTF-16 upstream of the
> JS engine, so the above actually works without any spec-language in
> ECMA-262 saying it should.)
>
> So based on a recent twitter/github exchange, gist recorded at
> https://gist.github.com/**1850768 <https://gist.github.com/1850768>, I
> would like to propose a variation on Allen's proposal that resolves both of
> these problems. Here are resolutions in reverse order:
>
> R2. No incompatible change without opt-in. If you hardcode as in Allen's
> example, don't opt in without changing your index, length, and char/code-at
> assumptions.
>
> Such opt-in cannot be a pragma since those have lexical scope and affect
> code, not the heap where strings and String.prototype methods live.
>
> We also wish to avoid exposing a "full Unicode" representation type and
> duplicated suite of the String static and prototype methods, as Java did.
> (We may well want UTF-N transcoding helpers; we certainly want ByteArray
> <-> UTF-8 transcoding APIs.)
>
> True, R2 implies there are two string primitive representations at most,
> or more likely "1.x" for some fraction .x. Say, a flag bit in the string
> header to distinguish JS's uint16-based indexing ("UCS-2") from
> non-O(1)-indexing UTF-16. Lots of non-observable implementation options
> here.
>
> Instead of any such *big* new observables, I propose a so-called "Big Red
> [opt-in] Switch" (BRS) on the side of a unit of VM isolation: specifically
> the global object.
>
> Why the global object? Because for many VMs, each global has its own heap
> or sub-heap ("compartment"), and all references outside that heap are to
> local proxies that copy from, or in the case of immutable data, reference
> the remote heap. Also because inter-compartment traffic is (we conjecture)
> infrequent enough to tolerate the proxy/copy overhead.
>
> For strings and String objects, such proxies would consult the remote
> heap's BRS setting and transcode indexed access, and .length gets,
> accordingly. It doesn't matter if the BRS is in the global or its String
> constructor or String.prototype, as the latter are unforgeably linked to
> the global.
>
> This means a script intent on comparing strings from two globals with
> different BRS settings could indeed tell that one discloses non-BMP
> char/codes, e.g. charCodeAt return values >= 0x10000. This is the *small*
> new observable I claim we can live with, because someone opted into it at
> least in one of the related global objects.
>
> Note that implementations such as Node.js can pre-set the BRS to "full
> Unicode" at startup. Embeddings that fully isolate each global and its
> reachable objects and strings pay no string-proxy or -copy overhead.
>
> R1. To keep compatibility with DOM APIs, the DOM glue used to mediate
> calls from JS to (typically) C++ would have to proxy or copy any strings
> containing non-BMP characters. Strings with only BMP characters would work
> as today.
>
> Note that we are dealing only in spec observables here. It doesn't matter
> whether the JS engine uses UTF-8 and the DOM UCS-2 (in which case there is
> already a transcoding penalty; IIRC WebKit libxml and libxslt use UTF-8 and
> so must transcode to interface with WebKit's DOM). The only issue at this
> boundary, I believe, is how indexing and .length work.
>
> Ok, there you have it: resolutions for both problems that killed the last
> assault on Castle '90s-JS.
>
> Implementations that use uint16 vectors as the character data
> representation type for both "UCS-2" and "UTF-16" string variants would
> probably want another flag bit per string header indicating whether, for
> the UTF-16 case, the string indeed contained any non-BMP characters. If
> not, no proxy/copy needed.
>
> Such implementations probably would benefit from string (primitive value)
> proxies not just copies, since the underlying uint16 vector could be shared
> by two different string headers with whatever metadata flag bits, etc., are
> needed to disclose different length values, access different methods from
> distinct globals' String.prototype objects, etc.
>
> We could certainly also work with the W3C to revise the DOM to check the
> BRS setting, if that is possible, to avoid this non-BMP-string proxy/copy
> overhead.
>
> How is the BRS configured? Again, not via a pragma, and not by imperative
> state update inside the language (mutating hidden BRS state at a given
> program point could leave strings created before mutation observably
> different from those created after, unless the implementation in effect
> scanned the local heap and wrapped or copied any non-BMP-char-bearing ones
> creatd before).
>
> The obvious way to express the BRS in HTML is a <meta> tag in document
> <head>, but I don't want to get hung up on this point. I do welcome expert
> guidance. Here is another W3C/WHATWG interaction point. For this reason I'm
> cc'ing public-script-coord.
>
> The upshot of this proposal is to get JS out of the '90s without a
> mandatory breaking change. With simple-enough opt-in expressed at
> coarse-enough boundaries so as not to impose high cost or unintended string
> type confusion bugs, the complexity is mostly borne by implementors, and at
> less than a 2x cost comparing string implementations (I think --
> demonstration required of course).
>
> In particular, Node.js can get modern at startup, and perhaps engines such
> as V8 as used in Node could even support compile-time (#ifdef) configury by
> which to support only full Unicode.
>
> Comments welcome.
>
> /be
>
> ______________________________**_________________
> es-discuss mailing list
> es-discuss@mozilla.org
> https://mail.mozilla.org/**listinfo/es-discuss<https://mail.mozilla.org/listinfo/es-discuss>
>