Re: Proper use of compositionstart and compositionupdate events for Android IMEs

Hey Ryan,

thanks so much for looking into this! This has been a pain point for a long
time and I'm glad we are finally getting some relief.

On Tue, Feb 20, 2018 at 10:58 PM, Ryan Landay <rlanday@chromium.org> wrote:

> Nearly all Android on-screen keyboards use the Android IME APIs for
> entering text:
> https://developer.android.com/reference/android/view/inputme
> thod/InputConnection.html
>
> For example, with Gboard, when you start typing a word, it will open a
> composition, update the composition for each letter typed, and then end the
> composition when you hit the space bar to end the word. If you tap on
> another word, Gboard will set that word as the active composition, even
> before you start actually typing. This uses the
> InputConnection#setComposingRegion() method to move the composition range
> to an existing piece of text:
>
> *https://developer.android.com/reference/android/view/inputmethod/InputConnection.html#setComposingRegion(int,%20int)
> <https://developer.android.com/reference/android/view/inputmethod/InputConnection.html#setComposingRegion(int,%20int)>*
>
>
> In Chrome 64 and earlier, we never fired compositionstart or
> compositionupdate events in response to setComposingRegion(). However, we
> _would_ actually move or create the composition region, and fire
> compositionend when that region was closed out. I landed a change for
> Chrome 65 that will fix this, so setComposingRegion() now fires
> compositionstart if no composition range is open, and compositionupdate
> regardless of whether or not one is already open.
>
> The UI Events spec says that for compositionstart:
>
> https://www.w3.org/TR/uievents/#event-type-compositionstart
> > Some implementations MAY populate the data attribute of the
> compositionstart event with the text currently selected in the document
> (for editing and replacement). Otherwise, the value of the data attribute
> MUST be the empty string.
>

I am not sure how this text ended up there, but that sounds more like a
description of what some implementations were doing already than was then
added to the spec than a helpful feature. It's not useful as one cannot
really know if the composition string is really empty or whether the UA
just opts for the second option, and secondly it seems useless because it
refers to the selection rather than the composition string.


>
> On Android, selection and composition are usually two different ranges.
> Tapping on a new word will cause Gboard to change the composition range to
> contain that word, but the selection will just be a caret selection
> wherever you tapped. So my Chrome change always sends the empty string for
> this event. However, I wanted web developers to actually know what text is
> in the composition at this point, so I'm also firing a compositionupdate
> immediately afterward.
>

Ok, but the composition is not actually changing? To me it sounds like this
will just be adding to the confusion. It would be better if we could
clarify that we don't send the current selection but instead the
composition string. One doesn't really need the selection, as there are
other ways to get that.

How about changing the text to something like: "A user agent MUST populate
the data attribute of the compositionstart event with the initial
composition string"

That would make the event more useful. However, event that would not do
away with the need of getting level 2 of the input events spec implemented.
Say for example you have this text:


"The summer|summer is hot." ( | = caret) And the compositionstart event
says that the initial composition string is "summer". Well -- which one is
it? The first one or the second one? Many times we can guess it by looking
at the selection and the initial composition string, but in cases like
these we won't know.

This is the reason we in level 2 of the Input Events spec as part of the
IME composition have a first cancelable beforeinput event of the
inputType "deleteByComposition" with a target range that points to exactly
the composition range that is being used as the initial composition string.



>
> Some developers from Basecamp (which develops at least two rich text
> editor web apps: Trix and Draft.js) were testing a beta of Chrome 65 and
> discovered that this change breaks their apps:
> https://bugs.chromium.org/p/chromium/issues/detail?id=812674
>
> They feel that it's now difficult to tell when a composition is actually
> inserting or replacing text versus when a keyboard is just moving the
> composition.
>

That is an interesting discussion you had. As an editor developer, I would
initially say that if a composition opens and closes but nothing changes
and there is nothing I really can do about the composition event that
changed nothing -- then why notify me about it all?

The question changes if there is something that can be done -- for example
I can place an opening candidate window. That seems like it may still be
some years into the future (or?) so then it may be difficult to tell us
JavaScript developers that we need to listen to these two events
(compositionstart/end) because in a few years time we can do something
in-between. But that is not that bad, as long as it's just start and end.

What seems like a bigger distraction is having an update event for the
composition when really nothing has changed. If you just have a
compositionstart and a compositionend and no compositionupdates inbetween,
at least it should be quite easy to figure out that nothing has actually
changed.



> It seems that they were previously making use of the fact that Chrome was
> not sending the compositionstart event for the latter case.
>

Right, but there are thousands such hacks that are passed around, trying to
get something workable out of all the various events coming in.


> I think there's some operation they want to do with the new text after a
> composition is closed. I feel like there should be a better way to do this
> (e.g. using the beforeinput or input events in conjunction with
> compositionend?). So I think the change we're making brings Chrome into
> better alignment with the spec, but I'm interested in getting feedback,
> especially from developers of rich text editing applications, as to whether
> the spec could be changed somehow to make it more useful. E.g., should we
> be able to send the initial composition text in the data field of the
> compositionstart event, even if it's not selected?
>
>
Definitely. Not only should you be "allowed" -- that should be an
obligation. Otherwise how are we to know what the meaning of an empty
string is? It will then just turn into: Sniff the browser, if you are on
Chrome you know that empty string means that it's actually empty whereas on
Firefox it means... .


>
> I also have a question about when fake key events with the 229 key code
> should be sent for composition updates. The spec says:
>
> https://www.w3.org/TR/uievents/#events-composition-key-events
> > During the composition session, keydown and keyup events MUST still be
> sent, and these events MUST have the isComposing attribute set to true.
>
> The language here is unclear to me and other Chrome developers about what
> should be done if the composition was changed in response to something
> other than a virtual or physical key press event.
>
> An example in a prior section shows a handwriting recognition IME
> apparently not sending keydown/keyup events (or are they just omitted from
> the example?):
> https://www.w3.org/TR/uievents/#events-composition-handwriting
>
>
> Right now, my code isn't sending key press events when a user taps on a
> word and Gboard moves the composition to a new word. This seems like the
> correct behavior (since there's no actual key being pressed and no text is
> actually being changed). However, I think the spec is generally unclear as
> to handle this case. Another issue in my opinion is that the spec for
> compositionupdate says:
>
> https://www.w3.org/TR/uievents/#event-type-compositionupdate
> > A user agent SHOULD dispatch this event during a composition session
> when a text composition system updates its active text passage with a new
> character, which is reflected in the string in data.
>
> However, it doesn't really say what to do if an IME is moving the
> composition to a new piece of text, or updating the composition by more
> than one character (both of these cases are extremely common).
>

In my test of various Android keyboards I just noticed inconsistent
behavior. On desktop with latin keyboard that don't rely on IME for input,
key events will likely continue to be important for quite a while. On
mobile this may also be the case for some specific keyboards that emit the
exact key event for the key and don't do IME (Hacker's keyboard does that I
believe), but in general these events seem less useful because JS cannot
really do much until the composition has ended anyway.


>
>
> In summary:
>
> - Are the existing APIs sufficient for web developers to interpret when
> Gboard is editing text vs. just moving the composition range?
>
> -  Does the behavior required by the spec as the Chrome editing engineers
> make sense for this case?
>
> - Whatever we decide, I think the spec should be made more clear about
> what should happen in these cases.
>
>
> Thanks,
> Ryan Landay
>
>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org