Re: Proper use of compositionstart and compositionupdate events for Android IMEs

On Wed, Feb 21, 2018 at 12:50 AM, Ryan Landay <rlanday@chromium.org> wrote:

> Are you thinking of cases the selection is in one word and there is a
> composition taking place there and then the user taps the screen
> which moves the selection somewhere entirely different outside the previous
> composition range?
>
>
> Yes, exactly.
>


Ok, in those cases, even the candidate window may have to find a new place
I guess, so it seems like it would be best if this could be treated as if
there was an entirely new composition, even if you really just exchange the
composition string.

Question: Between such a "fake" compositionend and compositionstart event,
would it be possible for JavaScript to manipulate the DOM? I mean usually
within a composition not much can be done, so the JavaScript will wait
until the composition is done with applying any changes. In a collaborative
setup there may even be changes coming in from collaborators that are held
back until the composition is done. So it would be good to have at least
interruptions in the composition process (such as when the composition
range changes) to be able to work on trying to merge local and remote
changes.


>
> In those cases I would say there should be a compositionend/start pair.
>
>
>
>
>
> On Feb 20, 2018, at 3:49 PM, Johannes Wilm <johannes@fiduswriter.org>
> wrote:
>
>
>
> On Wed, Feb 21, 2018 at 12:22 AM, Ryan Landay <rlanday@chromium.org>
> wrote:
>
>> How about changing the text to something like: "A user agent MUST
>> populate the data attribute of the compositionstart event with the
>> initial composition string"
>>
>>
>> I like this idea. One thing I think we still need though is a way to tell
>> the difference between an IME resetting the composition on a new range of
>> text (with a composition already open somewhere else) vs. changing the text
>> in the existing composition.
>>
> Perhaps this could be resolved by a requirement that the browser send a
>> compositionend/compositionstart pair for this case instead of sending
>> compositionupdate?
>>
>
>
> Are you thinking of cases the selection is in one word and there is a
> composition taking place there and then the user taps the screen which
> moves the selection somewhere entirely different outside the previous
> composition range?
>
> In those cases I would say there should be a compositionend/start pair.
>
>
>
>> Another option would be some sort of flag on compositionupdate, but I
>> think that might be unnecessary complexity.
>>
>> That would make the event more useful. However, event that would not do
>> away with the need of getting level 2 of the input events spec implemented.
>>
>>
>> I suspect the Chrome team might reconsider Input Events Level 2 at some
>> point. I agree we need better APIs around composition ranges. Unfortunately
>> I don’t think we have any big IME-related projects planned at the moment. I
>> was just trying to clean up what we already have when I made this change.
>>
>
> Right. I wasn't expecting you guys to do this right now. I was just making
> sure we all realize there is something in level 2 which would resolve some
> issues related to this.
>
>
>> On Feb 20, 2018, at 3:04 PM, Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>>
>> Hey Ryan,
>>
>> thanks so much for looking into this! This has been a pain point for a
>> long time and I'm glad we are finally getting some relief.
>>
>> On Tue, Feb 20, 2018 at 10:58 PM, Ryan Landay <rlanday@chromium.org>
>> wrote:
>>
>>> Nearly all Android on-screen keyboards use the Android IME APIs for
>>> entering text:
>>> https://developer.android.com/reference/android/view/inputme
>>> thod/InputConnection.html
>>>
>>> For example, with Gboard, when you start typing a word, it will open a
>>> composition, update the composition for each letter typed, and then end the
>>> composition when you hit the space bar to end the word. If you tap on
>>> another word, Gboard will set that word as the active composition, even
>>> before you start actually typing. This uses the
>>> InputConnection#setComposingRegion() method to move the composition
>>> range to an existing piece of text:
>>>
>>> *https://developer.android.com/reference/android/view/inputmethod/InputConnection.html#setComposingRegion(int,%20int)
>>> <https://developer.android.com/reference/android/view/inputmethod/InputConnection.html#setComposingRegion(int,%20int)>*
>>>
>>>
>>> In Chrome 64 and earlier, we never fired compositionstart or
>>> compositionupdate events in response to setComposingRegion(). However, we
>>> _would_ actually move or create the composition region, and fire
>>> compositionend when that region was closed out. I landed a change for
>>> Chrome 65 that will fix this, so setComposingRegion() now fires
>>> compositionstart if no composition range is open, and compositionupdate
>>> regardless of whether or not one is already open.
>>>
>>> The UI Events spec says that for compositionstart:
>>>
>>> https://www.w3.org/TR/uievents/#event-type-compositionstart
>>> > Some implementations MAY populate the data attribute of the
>>> compositionstart event with the text currently selected in the document
>>> (for editing and replacement). Otherwise, the value of the data attribute
>>> MUST be the empty string.
>>>
>>
>> I am not sure how this text ended up there, but that sounds more like a
>> description of what some implementations were doing already than was then
>> added to the spec than a helpful feature. It's not useful as one cannot
>> really know if the composition string is really empty or whether the UA
>> just opts for the second option, and secondly it seems useless because it
>> refers to the selection rather than the composition string.
>>
>>
>>>
>>> On Android, selection and composition are usually two different ranges.
>>> Tapping on a new word will cause Gboard to change the composition range to
>>> contain that word, but the selection will just be a caret selection
>>> wherever you tapped. So my Chrome change always sends the empty string for
>>> this event. However, I wanted web developers to actually know what text is
>>> in the composition at this point, so I'm also firing a compositionupdate
>>> immediately afterward.
>>>
>>
>> Ok, but the composition is not actually changing? To me it sounds like
>> this will just be adding to the confusion. It would be better if we could
>> clarify that we don't send the current selection but instead the
>> composition string. One doesn't really need the selection, as there are
>> other ways to get that.
>>
>> How about changing the text to something like: "A user agent MUST
>> populate the data attribute of the compositionstart event with the initial
>> composition string"
>>
>> That would make the event more useful. However, event that would not do
>> away with the need of getting level 2 of the input events spec implemented.
>> Say for example you have this text:
>>
>>
>> "The summer|summer is hot." ( | = caret) And the compositionstart event
>> says that the initial composition string is "summer". Well -- which one is
>> it? The first one or the second one? Many times we can guess it by looking
>> at the selection and the initial composition string, but in cases like
>> these we won't know.
>>
>> This is the reason we in level 2 of the Input Events spec as part of the
>> IME composition have a first cancelable beforeinput event of the
>> inputType "deleteByComposition" with a target range that points to
>> exactly the composition range that is being used as the initial composition
>> string.
>>
>>
>>
>>>
>>> Some developers from Basecamp (which develops at least two rich text
>>> editor web apps: Trix and Draft.js) were testing a beta of Chrome 65 and
>>> discovered that this change breaks their apps:
>>> https://bugs.chromium.org/p/chromium/issues/detail?id=812674
>>>
>>> They feel that it's now difficult to tell when a composition is actually
>>> inserting or replacing text versus when a keyboard is just moving the
>>> composition.
>>>
>>
>> That is an interesting discussion you had. As an editor developer, I
>> would initially say that if a composition opens and closes but nothing
>> changes and there is nothing I really can do about the composition event
>> that changed nothing -- then why notify me about it all?
>>
>> The question changes if there is something that can be done -- for
>> example I can place an opening candidate window. That seems like it may
>> still be some years into the future (or?) so then it may be difficult to
>> tell us JavaScript developers that we need to listen to these two events
>> (compositionstart/end) because in a few years time we can do something
>> in-between. But that is not that bad, as long as it's just start and end.
>>
>> What seems like a bigger distraction is having an update event for the
>> composition when really nothing has changed. If you just have a
>> compositionstart and a compositionend and no compositionupdates inbetween,
>> at least it should be quite easy to figure out that nothing has actually
>> changed.
>>
>>
>>
>>> It seems that they were previously making use of the fact that Chrome
>>> was not sending the compositionstart event for the latter case.
>>>
>>
>> Right, but there are thousands such hacks that are passed around, trying
>> to get something workable out of all the various events coming in.
>>
>>
>>> I think there's some operation they want to do with the new text after a
>>> composition is closed. I feel like there should be a better way to do this
>>> (e.g. using the beforeinput or input events in conjunction with
>>> compositionend?). So I think the change we're making brings Chrome into
>>> better alignment with the spec, but I'm interested in getting feedback,
>>> especially from developers of rich text editing applications, as to whether
>>> the spec could be changed somehow to make it more useful. E.g., should we
>>> be able to send the initial composition text in the data field of the
>>> compositionstart event, even if it's not selected?
>>>
>>>
>> Definitely. Not only should you be "allowed" -- that should be an
>> obligation. Otherwise how are we to know what the meaning of an empty
>> string is? It will then just turn into: Sniff the browser, if you are on
>> Chrome you know that empty string means that it's actually empty whereas on
>> Firefox it means... .
>>
>>
>>>
>>> I also have a question about when fake key events with the 229 key code
>>> should be sent for composition updates. The spec says:
>>>
>>> https://www.w3.org/TR/uievents/#events-composition-key-events
>>> > During the composition session, keydown and keyup events MUST still be
>>> sent, and these events MUST have the isComposing attribute set to true.
>>>
>>> The language here is unclear to me and other Chrome developers about
>>> what should be done if the composition was changed in response to something
>>> other than a virtual or physical key press event.
>>>
>>> An example in a prior section shows a handwriting recognition IME
>>> apparently not sending keydown/keyup events (or are they just omitted from
>>> the example?):
>>> https://www.w3.org/TR/uievents/#events-composition-handwriting
>>>
>>>
>>> Right now, my code isn't sending key press events when a user taps on a
>>> word and Gboard moves the composition to a new word. This seems like the
>>> correct behavior (since there's no actual key being pressed and no text is
>>> actually being changed). However, I think the spec is generally unclear as
>>> to handle this case. Another issue in my opinion is that the spec for
>>> compositionupdate says:
>>>
>>> https://www.w3.org/TR/uievents/#event-type-compositionupdate
>>> > A user agent SHOULD dispatch this event during a composition session
>>> when a text composition system updates its active text passage with a new
>>> character, which is reflected in the string in data.
>>>
>>> However, it doesn't really say what to do if an IME is moving the
>>> composition to a new piece of text, or updating the composition by more
>>> than one character (both of these cases are extremely common).
>>>
>>
>> In my test of various Android keyboards I just noticed inconsistent
>> behavior. On desktop with latin keyboard that don't rely on IME for input,
>> key events will likely continue to be important for quite a while. On
>> mobile this may also be the case for some specific keyboards that emit the
>> exact key event for the key and don't do IME (Hacker's keyboard does that I
>> believe), but in general these events seem less useful because JS cannot
>> really do much until the composition has ended anyway.
>>
>>
>>>
>>>
>>> In summary:
>>>
>>> - Are the existing APIs sufficient for web developers to interpret when
>>> Gboard is editing text vs. just moving the composition range?
>>>
>>> -  Does the behavior required by the spec as the Chrome editing
>>> engineers make sense for this case?
>>>
>>> - Whatever we decide, I think the spec should be made more clear about
>>> what should happen in these cases.
>>>
>>>
>>> Thanks,
>>> Ryan Landay
>>>
>>>
>>
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>>
>>
>
>
> --
> Johannes Wilm
> Fidus Writer
> http://www.fiduswriter.org
>
>
>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org