Re: Proper use of compositionstart and compositionupdate events for Android IMEs

On Wed, Feb 21, 2018 at 12:22 AM, Ryan Landay <rlanday@chromium.org> wrote:

> How about changing the text to something like: "A user agent MUST populate
> the data attribute of the compositionstart event with the
> initial composition string"
>
>
> I like this idea. One thing I think we still need though is a way to tell
> the difference between an IME resetting the composition on a new range of
> text (with a composition already open somewhere else) vs. changing the text
> in the existing composition.
>
Perhaps this could be resolved by a requirement that the browser send a
> compositionend/compositionstart pair for this case instead of sending
> compositionupdate?
>


Are you thinking of cases the selection is in one word and there is a
composition taking place there and then the user taps the screen which
moves the selection somewhere entirely different outside the previous
composition range?

In those cases I would say there should be a compositionend/start pair.



> Another option would be some sort of flag on compositionupdate, but I
> think that might be unnecessary complexity.
>
> That would make the event more useful. However, event that would not do
> away with the need of getting level 2 of the input events spec implemented.
>
>
> I suspect the Chrome team might reconsider Input Events Level 2 at some
> point. I agree we need better APIs around composition ranges. Unfortunately
> I don’t think we have any big IME-related projects planned at the moment. I
> was just trying to clean up what we already have when I made this change.
>

Right. I wasn't expecting you guys to do this right now. I was just making
sure we all realize there is something in level 2 which would resolve some
issues related to this.


> On Feb 20, 2018, at 3:04 PM, Johannes Wilm <johannes@fiduswriter.org>
> wrote:
>
> Hey Ryan,
>
> thanks so much for looking into this! This has been a pain point for a
> long time and I'm glad we are finally getting some relief.
>
> On Tue, Feb 20, 2018 at 10:58 PM, Ryan Landay <rlanday@chromium.org>
> wrote:
>
>> Nearly all Android on-screen keyboards use the Android IME APIs for
>> entering text:
>> https://developer.android.com/reference/android/view/inputme
>> thod/InputConnection.html
>>
>> For example, with Gboard, when you start typing a word, it will open a
>> composition, update the composition for each letter typed, and then end the
>> composition when you hit the space bar to end the word. If you tap on
>> another word, Gboard will set that word as the active composition, even
>> before you start actually typing. This uses the
>> InputConnection#setComposingRegion() method to move the composition
>> range to an existing piece of text:
>>
>> *https://developer.android.com/reference/android/view/inputmethod/InputConnection.html#setComposingRegion(int,%20int)
>> <https://developer.android.com/reference/android/view/inputmethod/InputConnection.html#setComposingRegion(int,%20int)>*
>>
>>
>> In Chrome 64 and earlier, we never fired compositionstart or
>> compositionupdate events in response to setComposingRegion(). However, we
>> _would_ actually move or create the composition region, and fire
>> compositionend when that region was closed out. I landed a change for
>> Chrome 65 that will fix this, so setComposingRegion() now fires
>> compositionstart if no composition range is open, and compositionupdate
>> regardless of whether or not one is already open.
>>
>> The UI Events spec says that for compositionstart:
>>
>> https://www.w3.org/TR/uievents/#event-type-compositionstart
>> > Some implementations MAY populate the data attribute of the
>> compositionstart event with the text currently selected in the document
>> (for editing and replacement). Otherwise, the value of the data attribute
>> MUST be the empty string.
>>
>
> I am not sure how this text ended up there, but that sounds more like a
> description of what some implementations were doing already than was then
> added to the spec than a helpful feature. It's not useful as one cannot
> really know if the composition string is really empty or whether the UA
> just opts for the second option, and secondly it seems useless because it
> refers to the selection rather than the composition string.
>
>
>>
>> On Android, selection and composition are usually two different ranges.
>> Tapping on a new word will cause Gboard to change the composition range to
>> contain that word, but the selection will just be a caret selection
>> wherever you tapped. So my Chrome change always sends the empty string for
>> this event. However, I wanted web developers to actually know what text is
>> in the composition at this point, so I'm also firing a compositionupdate
>> immediately afterward.
>>
>
> Ok, but the composition is not actually changing? To me it sounds like
> this will just be adding to the confusion. It would be better if we could
> clarify that we don't send the current selection but instead the
> composition string. One doesn't really need the selection, as there are
> other ways to get that.
>
> How about changing the text to something like: "A user agent MUST
> populate the data attribute of the compositionstart event with the initial
> composition string"
>
> That would make the event more useful. However, event that would not do
> away with the need of getting level 2 of the input events spec implemented.
> Say for example you have this text:
>
>
> "The summer|summer is hot." ( | = caret) And the compositionstart event
> says that the initial composition string is "summer". Well -- which one is
> it? The first one or the second one? Many times we can guess it by looking
> at the selection and the initial composition string, but in cases like
> these we won't know.
>
> This is the reason we in level 2 of the Input Events spec as part of the
> IME composition have a first cancelable beforeinput event of the inputType "deleteByComposition"
> with a target range that points to exactly the composition range that is
> being used as the initial composition string.
>
>
>
>>
>> Some developers from Basecamp (which develops at least two rich text
>> editor web apps: Trix and Draft.js) were testing a beta of Chrome 65 and
>> discovered that this change breaks their apps:
>> https://bugs.chromium.org/p/chromium/issues/detail?id=812674
>>
>> They feel that it's now difficult to tell when a composition is actually
>> inserting or replacing text versus when a keyboard is just moving the
>> composition.
>>
>
> That is an interesting discussion you had. As an editor developer, I would
> initially say that if a composition opens and closes but nothing changes
> and there is nothing I really can do about the composition event that
> changed nothing -- then why notify me about it all?
>
> The question changes if there is something that can be done -- for example
> I can place an opening candidate window. That seems like it may still be
> some years into the future (or?) so then it may be difficult to tell us
> JavaScript developers that we need to listen to these two events
> (compositionstart/end) because in a few years time we can do something
> in-between. But that is not that bad, as long as it's just start and end.
>
> What seems like a bigger distraction is having an update event for the
> composition when really nothing has changed. If you just have a
> compositionstart and a compositionend and no compositionupdates inbetween,
> at least it should be quite easy to figure out that nothing has actually
> changed.
>
>
>
>> It seems that they were previously making use of the fact that Chrome was
>> not sending the compositionstart event for the latter case.
>>
>
> Right, but there are thousands such hacks that are passed around, trying
> to get something workable out of all the various events coming in.
>
>
>> I think there's some operation they want to do with the new text after a
>> composition is closed. I feel like there should be a better way to do this
>> (e.g. using the beforeinput or input events in conjunction with
>> compositionend?). So I think the change we're making brings Chrome into
>> better alignment with the spec, but I'm interested in getting feedback,
>> especially from developers of rich text editing applications, as to whether
>> the spec could be changed somehow to make it more useful. E.g., should we
>> be able to send the initial composition text in the data field of the
>> compositionstart event, even if it's not selected?
>>
>>
> Definitely. Not only should you be "allowed" -- that should be an
> obligation. Otherwise how are we to know what the meaning of an empty
> string is? It will then just turn into: Sniff the browser, if you are on
> Chrome you know that empty string means that it's actually empty whereas on
> Firefox it means... .
>
>
>>
>> I also have a question about when fake key events with the 229 key code
>> should be sent for composition updates. The spec says:
>>
>> https://www.w3.org/TR/uievents/#events-composition-key-events
>> > During the composition session, keydown and keyup events MUST still be
>> sent, and these events MUST have the isComposing attribute set to true.
>>
>> The language here is unclear to me and other Chrome developers about what
>> should be done if the composition was changed in response to something
>> other than a virtual or physical key press event.
>>
>> An example in a prior section shows a handwriting recognition IME
>> apparently not sending keydown/keyup events (or are they just omitted from
>> the example?):
>> https://www.w3.org/TR/uievents/#events-composition-handwriting
>>
>>
>> Right now, my code isn't sending key press events when a user taps on a
>> word and Gboard moves the composition to a new word. This seems like the
>> correct behavior (since there's no actual key being pressed and no text is
>> actually being changed). However, I think the spec is generally unclear as
>> to handle this case. Another issue in my opinion is that the spec for
>> compositionupdate says:
>>
>> https://www.w3.org/TR/uievents/#event-type-compositionupdate
>> > A user agent SHOULD dispatch this event during a composition session
>> when a text composition system updates its active text passage with a new
>> character, which is reflected in the string in data.
>>
>> However, it doesn't really say what to do if an IME is moving the
>> composition to a new piece of text, or updating the composition by more
>> than one character (both of these cases are extremely common).
>>
>
> In my test of various Android keyboards I just noticed inconsistent
> behavior. On desktop with latin keyboard that don't rely on IME for input,
> key events will likely continue to be important for quite a while. On
> mobile this may also be the case for some specific keyboards that emit the
> exact key event for the key and don't do IME (Hacker's keyboard does that I
> believe), but in general these events seem less useful because JS cannot
> really do much until the composition has ended anyway.
>
>
>>
>>
>> In summary:
>>
>> - Are the existing APIs sufficient for web developers to interpret when
>> Gboard is editing text vs. just moving the composition range?
>>
>> -  Does the behavior required by the spec as the Chrome editing engineers
>> make sense for this case?
>>
>> - Whatever we decide, I think the spec should be made more clear about
>> what should happen in these cases.
>>
>>
>> Thanks,
>> Ryan Landay
>>
>>
>
>
> --
> Johannes Wilm
> Fidus Writer
> http://www.fiduswriter.org
>
>
>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org