Re: New full Unicode for ES6 idea

Second part: the BRS.

I'm wondering how development and deployment of existing full-Unicode software will play out in the presence of a Big Red Switch. Maybe I'm blind and there are ways to simplify the process, but this is how I imagine it.

Let's start with a bit of code that currently supports full Unicode by hacking around ECMAScript's limitations:
https://github.com/roozbehp/yui3-gallery/blob/master/src/gallery-intl-bidi/js/intl-bidi.js

To support applications running in a BRS-on environment, Roozbeh would have to create a parallel version of the module that (a) takes advantage of regular expressions that finally support supplementary characters and (b) uses the new Unicode escape syntax instead of the old one. The parallel version has to be completely separate because a BRS-on environment would reject the old Unicode escapes and an ES5/BRS-off environment would reject the new Unicode escapes.

To get the code tested, he also has to create a parallel version of the test cases. The parallel version would be functionally identical but set up a BRS-on environment and use the new Unicode escape syntax instead of the old one. The parallel version has to be completely separate because a BRS-on environment would reject the old Unicode escapes and an ES5/BRS-off environment would reject the new Unicode escapes. Fortunately the test cases are simple.

Then he has to figure out how the two separate versions of the module will get loaded by clients. It's a YUI module, and the YUI loader already has the ability to look at several parameters to figure out what to load (minimized vs. debug version, localized resource bundles, etc.), so maybe the BRS should be another parameter? But the YUI team has a long to-do list, so in the meantime the module gets two separate names, and the client has to figure out which one to request.

The first client picking up the new version is another, bigger library. As a library it doesn't control the BRS, so it has to be able to run with both BRS-on and BRS-off. So it has to check the BRS and load the appropriate version of the intl-bidi module at runtime. This means, it also has to be tested in both environments. Its test cases are not simple. So now it needs modifications to the test framework to run the test suite twice, once with BRS-on and once with BRS-off.

An application using the library and thus the intl-bidi module decides to take the plunge and switch to BRS-on. It doesn't do text processing itself (that's what libraries are for), and it doesn't use Unicode escapes, so no code changes. But when it throws the switch, exceptions get thrown. It turns out that 3 of the 50 JavaScript files loaded during startup use old Unicode escapes. One of them seems to do something that might affect supplementary characters; for the other two apparently the developers just felt safer escaping all non-ASCII characters. The developers of the application don't actually know anything about the scripts - they got loaded indirectly by apps, ads, and analytics software used by the application. The developers try to find out whom they'll have to educate about the BRS to get this resolved.

OK - migrations are hard. But so far most participants have only seen additional work, no benefits. How long will this take? When will it end? When will browsers make BRS-on the default, let alone eliminate the switch? When can Roozbeh abandon his original version? Where's the blue button?

The thing to keep in mind is that most code doesn't need to know anything about supplementary characters. The beneficiaries of the switch are only the implementors of functions that do need to know, and even they won't really benefit until the switch is permanently on (at least for all their clients). It seems the switch puts a new burden on many that so far have been rightfully oblivious to supplementary characters.

Norbert



On Feb 19, 2012, at 0:33 , Brendan Eich wrote:

[snip]

> Allen's strawman from last year, http://wiki.ecmascript.org/doku.php?id=strawman:support_full_unicode_in_strings, proposed a brute-force change to support full Unicode (albeit with too many hex digits allowed in "\u{...}"), observing that "There are very few places where the ECMAScript specification has actual dependencies upon the size of individual characters so the compatibility impact of supporting full Unicode is quite small." But two problems remained:
> 
> P1. As Allen wrote, "There is a larger impact on actual implementations", and no implementors that I can recall were satisfied that the cost was acceptable. It might be, we just didn't know, and there are enough signs of high cost to create this concern.
> 
> P2. The change is not backward compatible. In JS today, one read a string s from somewhere and hard-code, e.g., s.indexOf("0xd800" to find part of a surrogate pair, then advance to the next-indexed uint16 unit and read the other half, then combine to compute some result. Such usage would break.
> 
> Example from Allen:
> 
> var c = "😁" // where the single character between the quotes is the Unicode character U+1f638
> 
> c.length == 2;
> c === "\ud83d\ude38"; //the two character UTF-16 encoding of 0x1f683
> c.charCodeAt(0) == 0xd83d;
> c.charCodeAt(1) == 0xd338;
> 
> (Allen points out how browsers, node.js, and other environments blindly handle UTF-8 or whatever incoming format recoding to UTF-16 upstream of the JS engine, so the above actually works without any spec-language in ECMA-262 saying it should.)
> 
> So based on a recent twitter/github exchange, gist recorded at https://gist.github.com/1850768, I would like to propose a variation on Allen's proposal that resolves both of these problems. Here are resolutions in reverse order:
> 
> R2. No incompatible change without opt-in. If you hardcode as in Allen's example, don't opt in without changing your index, length, and char/code-at assumptions.
> 
> Such opt-in cannot be a pragma since those have lexical scope and affect code, not the heap where strings and String.prototype methods live.
> 
> We also wish to avoid exposing a "full Unicode" representation type and duplicated suite of the String static and prototype methods, as Java did. (We may well want UTF-N transcoding helpers; we certainly want ByteArray <-> UTF-8 transcoding APIs.)
> 
> True, R2 implies there are two string primitive representations at most, or more likely "1.x" for some fraction .x. Say, a flag bit in the string header to distinguish JS's uint16-based indexing ("UCS-2") from non-O(1)-indexing UTF-16. Lots of non-observable implementation options here.
> 
> Instead of any such *big* new observables, I propose a so-called "Big Red [opt-in] Switch" (BRS) on the side of a unit of VM isolation: specifically the global object.
> 
> Why the global object? Because for many VMs, each global has its own heap or sub-heap ("compartment"), and all references outside that heap are to local proxies that copy from, or in the case of immutable data, reference the remote heap. Also because inter-compartment traffic is (we conjecture) infrequent enough to tolerate the proxy/copy overhead.
> 
> For strings and String objects, such proxies would consult the remote heap's BRS setting and transcode indexed access, and .length gets, accordingly. It doesn't matter if the BRS is in the global or its String constructor or String.prototype, as the latter are unforgeably linked to the global.
> 
> This means a script intent on comparing strings from two globals with different BRS settings could indeed tell that one discloses non-BMP char/codes, e.g. charCodeAt return values >= 0x10000. This is the *small* new observable I claim we can live with, because someone opted into it at least in one of the related global objects.
> 
> Note that implementations such as Node.js can pre-set the BRS to "full Unicode" at startup. Embeddings that fully isolate each global and its reachable objects and strings pay no string-proxy or -copy overhead.
> 
> R1. To keep compatibility with DOM APIs, the DOM glue used to mediate calls from JS to (typically) C++ would have to proxy or copy any strings containing non-BMP characters. Strings with only BMP characters would work as today.
> 
> Note that we are dealing only in spec observables here. It doesn't matter whether the JS engine uses UTF-8 and the DOM UCS-2 (in which case there is already a transcoding penalty; IIRC WebKit libxml and libxslt use UTF-8 and so must transcode to interface with WebKit's DOM). The only issue at this boundary, I believe, is how indexing and .length work.
> 
> Ok, there you have it: resolutions for both problems that killed the last assault on Castle '90s-JS.
> 
> Implementations that use uint16 vectors as the character data representation type for both "UCS-2" and "UTF-16" string variants would probably want another flag bit per string header indicating whether, for the UTF-16 case, the string indeed contained any non-BMP characters. If not, no proxy/copy needed.
> 
> Such implementations probably would benefit from string (primitive value) proxies not just copies, since the underlying uint16 vector could be shared by two different string headers with whatever metadata flag bits, etc., are needed to disclose different length values, access different methods from distinct globals' String.prototype objects, etc.
> 
> We could certainly also work with the W3C to revise the DOM to check the BRS setting, if that is possible, to avoid this non-BMP-string proxy/copy overhead.
> 
> How is the BRS configured? Again, not via a pragma, and not by imperative state update inside the language (mutating hidden BRS state at a given program point could leave strings created before mutation observably different from those created after, unless the implementation in effect scanned the local heap and wrapped or copied any non-BMP-char-bearing ones creatd before).
> 
> The obvious way to express the BRS in HTML is a <meta> tag in document <head>, but I don't want to get hung up on this point. I do welcome expert guidance. Here is another W3C/WHATWG interaction point. For this reason I'm cc'ing public-script-coord.
> 
> The upshot of this proposal is to get JS out of the '90s without a mandatory breaking change. With simple-enough opt-in expressed at coarse-enough boundaries so as not to impose high cost or unintended string type confusion bugs, the complexity is mostly borne by implementors, and at less than a 2x cost comparing string implementations (I think -- demonstration required of course).
> 
> In particular, Node.js can get modern at startup, and perhaps engines such as V8 as used in Node could even support compile-time (#ifdef) configury by which to support only full Unicode.
> 
> Comments welcome.
> 
> /be
> 
> _______________________________________________
> es-discuss mailing list
> es-discuss@mozilla.org
> https://mail.mozilla.org/listinfo/es-discuss

Received on Thursday, 23 February 2012 09:24:44 UTC