Re: 2 Proposals for Minimum Viable InputEvent

On Tue, Feb 7, 2017 at 12:23 AM, Alexandre Elias <aelias@google.com> wrote:

> Hi, I'm the maintainer of IME on Android.  I'll be happy to attend the
> conference call next week.  As you've surmised, Android does need to
> perform a DOM change to merge two nodes "later".  However, I disagree that
> those 5 steps are somehow needed.  Our plan is to send a single
> insertCompositionText beforeinput event instead of 5 events.
>

Nice to meet you Alexandre!

It's a bit unfrotunate that you haven't been able to participate in the
last 3 years of discussions about this subject (I think some people have
even been discussing this for more than 8 years), and it seems that the
mailing list isn't as alive any more as it once used to be, but I'll do my
best to give the reasoning why we came to the conclusion that these events
are needed. I would also recommend consulting the various open and closed
issues on github and to consult the archive of the mailing list. Also,
various Google people have been behind a lot if not most of the various
various concepts, so they can probably give you more background about this
as well.


>
> Arguments why each is not needed:
>
>  > 1. deleteByComposition --This step is only needed when recomposing. A
> composed word can go across element boundaries, so this may delete more
> than just contents of a text node. (fx " He<b>ll</b><i>o fish</i>!"). This
> step needs to be executed at the moment your Android IME changes the DOM,
> whether this is at the moment the user taps the word and the underline
> appears or at a later stage.
>
> There's no need for a separate "delete" event, because
> insertion-to-replace-range is general enough to represent a deletion range
> as well.  I prefer to reserve "delete" events exclusively for intended IME
> deletions (e.g. select word and press backspace).
>

It is always the same event (the beforeinput event). The thing that is
differing on each one of them is the attribute "inputType".

The point of the beforeinput event is that instead of creating editors
based on a mixture of monitoring what is actually happening (for example
with mutation observers, diffing the dom every so often, etc.) and what the
user is doing (for example by listening to key click events), to let
JavaScript create editors that are based on what intentions the user has.
So for example, instead of analyzing a DOM-change and from that to
concluding that the user probably tried to make a certain part of the text
bold, we have an event that tells us directly that "the user intends to
make range X bold". The JS editor then decides what to do with that
information.

Due to this semantic nature of the inputTypes, seemingly similar events in
terms of DOM-change, can have very different meanings semantically.

In this case, the user never asked to delete a range. The user asked to
edit a specific word. There is a very concrete difference between the two,
because if the deletion is due to the beginning of a composition and not
because the user asked for it, the JS may attempt to recreate styling that
is lost in this initial deletion when the composition is finished. So we
need to distinguish between deletion because the user wants to delete, and
deletion because the IME can not handle partially styled words.

But I am a little confused: From the proposal Chong sent in, it sounded
like you just wouldn't have any beforeinput event for this initial
deletion. That would create a range of different problems, such as:

* The more advanced editors today are moving toward a model where they hold
a json version of the document which is separate from the DOM. They only
make adjustments to the DOM when needed, and if the user changes something
about the DOM, they make adjustments to the  json document. The json
version is the authoritative copy of the document. For this to work, they
need to know about any change to the DOM. If you make user-initiated DOM
changes that are not causing any beforeinput event at all, the beforeinput
event is somewhat useless, aand they will have to continue to monitor user
initiated DOM-changes in other ways.

* In a collaborative editor, if you make a change to the DOM of one user,
and the JS is not being made aware of this change, then it won't notify the
collaborators of this change, and so the collaborator will ahve different
versions of the document. Again, the point of beforeinput events will be
gone. JS develoeprs will use mutation observers or some other technology
instead.




>
> > 2. insertCompositionText -- Also this step only needs to happen at the
> moment only for recompositions. It reinserts the word from step one, but
> this time as plaintext in one element (fx "Hello" => "<i>Hello fish</i>")
>
> On Android because of the "late" timing, this merge operation would happen
> in the same message loop task as step 3, so there is no need for this
> event.  We prefer to go directly from "He<b>ll</b><i>o fish</i>" ->
> "<i>Help<i>".
>


Yes, in a perfect world, where all IME-related events are cancelable, this
could be done and would be preferable, as for example Yosin has pointed out
( https://github.com/w3c/input-events/issues/45#issuecomment-261427648 ).
We had a long discussion during the fall of 2015 about this, and among
other ones several developers from Chromium eventually convinced us here
that IME-events cannot be guaranteed to be cancelable. Even moving the
caret to somewhere else during the composition may somehow crash the IME
(hwo this is possible is a bit of a mystery to me as a JS developer).

One could then say ok, combine 1 and 2 and make them both non-cancelable.
The problem with that is that the browser doesn't always know the seamntic
meaning of the various DOM elements used within a document and it may end
up making decisions of what to delete that are problematic.



> However, rniwa@apple.com has told us that on OS X, the standard platform
> behavior is to merge the nodes at time of composition start.  Therefore, on
> OS X step 2 is still needed and there would be 2 beforeinput events in
> total, but on Android there would be only 1.  This represents a substantive
> platform difference that I think it makes sense to allow to differ in the
> spec.  I don't think we should try to artificially synchronize the
> platforms by always sending 2 events -- on Android this would simply result
> in unnecessary work and side effects that would never be seen by the user.
>

I think you should instead consider making this two events on Android.
Android is really great, but text editing in Chrome on Android is really
problematic especially when it comes to IME recomposition, as I am sure you
are aware. I have tried various keyboard, and but even on mainstream sites
such as Facebook, using a Google-provided keyboard, recomposition tends to
just mess up entire sentences, throw random words around, deleting or
merging words that are close to what one is trying to recompose, etc. .
Speed is not as much a concern anyway because it all depends on the human
who is interacting with it, but in addition these really major issues mean
that recomposition as it exists on Android as of today is not very useful.
In addition there are the various issues that JS developers have complained
about for a while: no compositionstart event, etc. . I am sure you know the
list of grievances with it all too well.

Not knowing the details, but I could imagine you can check for event
listeners, and only if those are present you make it two events.

But as an alternative, and to try to get a bit closer to both your and
Yosin's position, how about this:  We make two IME models, one for IMEs
where the beforeinput events can be cancelled in which step 1 and 2 are
merged and one where they cannot be cancelled where they are separate.
Eventually we want all beforeinput events to be cancellable, and this will
generally be nicer to work with for JS developers, but for now this would
mean that it could work with your existing way of doing things on Android,
and we don't force others to make IME cancellable. Of coruse this would
require for IME beforeinput events to be cancellable on Android/Chromium,
but from Yosin's comment above I udnerstand that this is the case, correct?




>
>
.
>
> > 3. insertCompositionText -- changes the composiiton (fx "p" with
> targetrange above "lo" => "Help"
>
> This one is needed.
>

agreed.

>
> > 4. deleteCompositionText -- removed (fx "help"): It removed the text
> from the underline and it may not be the entire composition.
>
> > 5. insertFromComposition -- adds "help" permanently, and gives the JS
> the opposrtunity to rebuild the complex HTML structure lost in step 1.
>
> Step 3 is already permanent.  The fact that there is still an active
> composition underline at the time of step 3 does not make it somehow
> temporary.  It has already been injected into the DOM and caused all side
> effects.  So the JS should rebuild its complex structure at time of event
> 3, and events 4/5 are redundant.
>


Again, in a perfect world where IME input is cancelable, this could
possibly be done. But this is not currently thr case. Right now JS editors
have to work this way with compositions: When a composition starts, they
temporarily pause most activities because they are not really allowed to
intervene. Then, once the composition is finished, they deal with the
finished text.  input. For example, in step 5 they may reapply some partial
styling that was lost in step 1. Or they could decide to change a word into
a tag. or a smiley. or apply some subject-specific text-replacement that
the browser doesn't need to understand. Or, as you have done in Google
Docs, you may want to divide up the finished composition string in
different elements for every line. I believe Ryosuke also mentioned the
case once where the individual characters of a title each are placed into
an individual span so that they can be positioned with CSS.

In conclusion: It seems that if you can make all the beforeinput events for
IME cancelable, you may possibly be able to cut it down to two events
without crippling the event: 1/2 when the composition starts, and 3/4/5
during the composition.  But will you be able to do that?



>
>>
>> The reason you state for the proposed change is that "On Android it’s
>> very common for IME to start composition whenever a cursor is moved onto a
>> word.  This is primarily intended to passively show an underline to the
>> user (to indicate what would be replaced *if* user selects an alternate
>> spelling)."
>>
>> If such tapping leads to actual DOM changes, then  the JS needs to know
>> about those(step 1/2). But if you don't change the DOM immediately and only
>> do so at a later stage when you "really" start the composition, then you
>> can also wait until that stage.
>>
>>
>> So these steps are needed on allplattforms, no matter whether or not you
>> change how IMEs work on Android.
>>
>>
>>>
>>> Adding aelias@ to make sure he is able to make the call on the 14th.
>>>
>>> dave.
>>>
>>> On Fri, Feb 3, 2017 at 1:33 AM, Johannes Wilm <johannes@fiduswriter.org>
>>> wrote:
>>>
>>>> Could we get the person or people who came up with these changes due to
>>>> Android to participate in the call on the 14th? I think it's very important
>>>> to have them invovled in this discussion.
>>>>
>>>> On Fri, Feb 3, 2017 at 10:21 AM, Johannes Wilm <
>>>> johannes@fiduswriter.org> wrote:
>>>>
>>>>>
>>>>> On Fri, Feb 3, 2017 at 10:18 AM, Johannes Wilm <
>>>>> johannes@fiduswriter.org> wrote:
>>>>> ...
>>>>>
>>>>>> Right now I cannot see any purpose of the beforeinput event with
>>>>>> these changes applied, but maybe I am missing something?
>>>>>>
>>>>>>
>>>>> I need to restarct that. I guess it could still be useful to stop
>>>>> native bold/italic buttons from making their own, non-controlled DOM
>>>>> changes. It's just for text input that it is irrelevant. And IME will
>>>>> continue to be a mess.
>>>>>
>>>>> --
>>>>> Johannes Wilm
>>>>> Fidus Writer
>>>>> http://www.fiduswriter.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Johannes Wilm
>>>> Fidus Writer
>>>> http://www.fiduswriter.org
>>>>
>>>
>>>
>>
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>
>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org