Re: Way forward and IME behavior speccing

On Sun, Oct 11, 2015 at 4:48 PM, Koji Ishii <kojiishi@gmail.com> wrote:

> I'm not sure we're moving towards to the right direction.
>
> I wasn't in the discussion, but what I remember was that the TF moved from
> ce=events to ce=typing because with events, JS has to handle all IMEs and
> composited characters, and Ben said that JS developers wanted to avoid that.
>

Ok, so for those new to the discussion, it went something like this:

Koji and Ben Petersen (at that time at Microsoft) were discussing ways of
making IME input transparent to the browser.

Ben left, and I joined this part of the conversation. The browser people
insisted that IMEs cannot be controlled in any way and that one cannot
cancel their insertion of a letter or otherwise. So I asked if at least we
could sandbox it. Koji mentioned we could extend text nodes to hold a
shadow root and do the editing process transparently inside of that.
Ryosuke (Apple) then said that was impossible. I then proposed we could
just use the lowest level element node, copy a text node and all its
siblings into it, then do the editing in that shadow root, all transparent
to the user. Ryosuke and Koji seemed to agree.

Then we met in Paris. Ojan (Google) said that doing this kind of shadow
root type handling was non-performant, but instead he offered that we could
write the spec so that moving the care to a newly created shadowDOM and do
the composition there could be handled in JavaScript. The editor people and
everyone present seemed happy with that. I wrote a small piece of
Javascript to try out whether this is actually working. It turns out it was
working on Chrome, But it was somewhat inconsistent, so I went and talked
to the Ui events group in the hope of getting standardization of it. Koji
and some others were there and once they heard what I was doing (moving the
caret during compositionstart), they wondered whether the spec should be
changed to forbid me from doing this. Among other things, a new and shiny
IME in Windows 10 is very fragile and apparently breaking this, so the spec
would possibly be needed to readjusted.

While it seems somewhat frustrating that there seems to be so little
understanding for why we need to spec anything at all, and why we need JS
editors, and why can't we just leave it all up for the browser, when
browsers have failed to advance in this area for 15 years and have no
comprehensive plan of their own to make it work, I guess these are the
types of conflicts one will run into any time one starts touching an area
that involves many people.


>
> If ce=typing needs to define and handle all IME related complications, it
> looks to me that we lost something. Not sure what we missed, but my
> preference is to discuss to find a way without defining everything related
> with IME.
>
> Do you have a list of issues why you need to handle IME, so that we could
> try to find a way without defining it?
>


Among the issues are:

1. Keep all changes atomic. This makes it much easier to handle in terms of
history and collaboration handling. One way of doing this is to sandbox the
IME, then figure out what the changes are when it is done (my approach).
Another approach is to just let the composition happen in the DOM and onc
eit is done, figure out what it has done and potentially role that change
back and then create an atomic change.

2. Just like editors don't want contentEditable to change the DOM, they
don't want IME to do it either. The editors want and need to be in charge
of what actually happens in the editor in the end. The "damage" that
browser in editable mode can do to the editing process can in most cases
either be directly canceled or otherwise immediately be "rolled back" by
diffing and patching the DOM. This is not the case for IME input, as the
composition process only can be interrupted on a limited and somewhat
unpredictable way. So at the very least one needs to let the IME finish the
entire composition, before one can roll the changes it has made to the DOM
back and instead change the DOM the way we really need to.

While IMEs mostly just manipulate the contents of text nodes, the future of
IMEs is apparently a lot more complex. As I believe you mentioned yourself,
Koji, IMEs on Android edit entire words, even if the word has more complex
DOM structures inside of it.


For example, say I have a language in which "2nd" is written as
"2<sup>a</sup>". So the user types "2" and "a", and the IME then
automatically inserts:

 2<sup>a</sup>

or because it is another browser it may decide to add this type of HTML
within the IME:

2<span style="font-size: 11px;font-face: Arial;vertical-align:sup;"
class="--Webkit-custom-property">a</span>

or some such thing. Because it's IME, one won't be able to cancel it or
change it, at least not before the entire composition is done.

So, to stop all of this, we need a clear specification of IMEs. And it
needs to be restrictive. There are a million things IMEs might do, but if
every IME does it's own thing, it's bound to just break most of the
internet and not to be of benefit to anyone. For example, I notice that
with the only multi language keyboard on Android 5.1 (Multiling O Keyboard)
with Chrome on standard sites such as Facebook, correcting previous
comments is just fundamentally broken. In essence, the argument for why IME
behavior needs to be standardized is the same argument for why anything
needs to be specced and standardized and why we cannot just have 4-5
different browsers, each doing it's own thing in a different way.



> /koji
>
> On Sat, Oct 10, 2015 at 12:41 AM, Johannes Wilm <johannes@fiduswriter.org>
> wrote:
>
>> Hey,
>> as you all will know, we had quite a productive F2F meeting in Paris in
>> connection with the CSSWG at the end of August.
>>
>> Unfortunately a few more issues appeared just after the meeting had
>> concluded, principally issues the browser makers found after concluding the
>> meeting that had not come up during the meeting itself. We hope to be able
>> to clarify all these points during the Wednesday of TPAC [1].
>>
>> The issues are:
>>
>> -- The name of the events (beforeInput/Input or beforeEdit/Edit). [2]
>> -- What is the complete list of edit operations for
>> beforeEdit/beforeInput? [3]
>> -- Complications related to IME input. [4]
>>
>> Particularly IME input seems to be an evergrowing problem. We already had
>> to move from cE=events to cE=typing, which is a odd mode in that it only
>> allows character input, and deletion, but deletion only during IME
>> composition. But now it seems that even that is not enough, that IME input
>> need to be able to arbitrarily change the DOM and that IME composition
>> cannot in any way be restricted. Also, IME is working differently on every
>> platform and with every IME engine and a lot of IME interfaces are
>> apparently constantly on the brink of breaking.
>>
>> To me this all sounds as if IME is like Flash, just that there are 30
>> different versions of it now and need to accommodate those + 150 future
>> versions whose functionality is not yet known . Ie - it's not a situation
>> that is viable in the long term. And because the usage of IMEs is expanding
>> (on Android everything is IME), the only way to kill the monster is to
>> start speccing the behavior of IMEs and to make sure they  all are
>> restricted in their behavior and that their behavior is standardized.
>>
>> The IME API [5] does not seem to talk about any of that. The question is:
>> Is this a document that still needs to be made, or is it already happening
>> somewhere else?
>>
>>
>> Btw, to me it seemed that one of the more important points is to sandbox
>> the IME [4], but as I understand Piotr (CK Editor), that is not necessarily
>> enough -- being able to cancel or modify individual composition operations
>> (including deletions) would be more important.
>>
>>
>>
>> [1]
>> https://www.w3.org/wiki/TPAC/2015/SessionIdeas#Text_editing_in_the_browser
>> [2] https://github.com/w3c/editing/issues/87
>> [3] https://github.com/w3c/editing/issues/79
>> [4] See for example
>> https://github.com/w3c/uievents/issues/5#issuecomment-137559461
>> [5] http://www.w3.org/TR/ime-api/
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>
>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org