Re: cancelability in level 1 of input-events spec

Thanks for taking the time to better understand Android behavior.

On Tue, Jun 27, 2017 at 8:19 AM, Johannes Wilm <mail@johanneswilm.org>
wrote:

> Hey again,
>
> I've been trying to understand the problem you guys have on Android a bit
> better and have done some testing with different keyboards on Android and
> reading up on what I could find of documentation of IME on Android.
>
> This mroning I consulted with Piotr Koszuliński from CKEditor and had
> earlier discussed with Marijn Haverbeke from ProseMirror specifically about
> contenteditable issues on Android. Then I did some testing myself.
>
> I may have misunderstood some things, so please enlighten me in case I
> should be wrong.
>
> In brief, it seems like:
>
> A. On Android, the right keyboard related events are sent only if the
> keyboard has implemented everything correctly.
> B. The standard, Google-provided keyboard, is not sending the rights
> events, and those events that are sent have incorrect information.
>

>From Chromium-team's point of view, any pattern of Java API calls that
Google Keyboard (or for that matter Samsung Keyboard, LG keyboard etc) does
is by definition "right", and something we have very little power to
change.  Chromium-team has no power to dictate to Android keyboard apps
whatever behavior we might subjectively think is "right".

However, in those cases where a problem is 100% due to how Chromium
interprets/mangles the Google Keyboard Java API calls to JS, I *can* do
something about it.

C. There may be some things that are defined by the browser about these
> events that is giving incorrect information to JS, given that all keyboard
> tested showed some common bugs.
> D. The Android IME interface allows the IME to edit the document in a few
> limited ways, none of which include the semantic editing operations which
> you have decided to make non-cancelable in Level 1 of the Input Events spec.
>
> ----
>
> *Testing: *
>
> I have looked at the Google Keyboard (Gboard), Fleksy Keyboard, Multiling
> O Keyboard and Hacker's Keyboard on contenteditable and operations:
>
> typing character within word
> backspace within word
> backspace between words
> backspace at beginning of paragraph (with another previous paragraph)
> selecting text from one paragraph to another and then typing a character
>
>
> Result for Google Keyboard:
>
> - On Android, no composition events (compositionstart, compositionupdate,
> compositionend) are triggered, no matter what. This looks like a bug.
>

I would indeed consider that a bug, but I cannot reproduce this
observation.  On this test page
<https://cdn.rawgit.com/w3c/uievents/gh-pages/tools/key-event-viewer.html>,
using Chrome Beta 60.0.3112.43 and GBoard Beta 6.3.30.159940404 on Galaxy
S7/NRD90M, I see the compositionStart/Update/End events just fine.  Let me
know what exact versions of all those things you observed this with.


> - The keydown event sometimes has isComposing = true, and sometimes
> isComposing = false, but this seems to have no influence over how it works.
> This looks like a bug.
>

It seems to accurately reflect whether a composing underline is currently
present.  I'm not sure that anything else is specified in terms of
influencing how it works.  In what sense is the current havior a bug?


> - The keydown events on Android claim they are cancelable, but
> event.preventDefault() does nothing. This looks like a bug.
>

Thanks.  I agree that is a bug and one that we're fully empowered to fix
within Chromium (at least to the extent of making cancelable accurately
false).  I filed http://crbug.com/737349 to track it.


> - The keydown events all have keyCode = 229, which is also what desktop
> browsers do during IME.
>

Yes.  The choice of 229, a number originating from MS Windows, isn't ideal
on Android because 229 is also mapped to LAST_CHANNEL (Android TV remote
key).  We are considering changing it to 0 keycode instead, as some other
desktop browsers do.


> - An exception is the Enter key, which does give a keyCode, and which can
> be canceled with event.preventDefault().
>

This is a known problem.  In Java InputConnection, Google Keyboard still
generates the usual commitText("\n") call for Enter, but Chromium was
forced to introduce a workaround for the Android WebView-based GMail native
app, which relies on the keyCode for its special inline reply break
behavior.  This is brittle and doesn't work properly in all cases, so we're
working with them to transition to a better approach, after which we hope
to remove this behavior.  Removal is tracked in http://crbug.com/579652


>
> Results for Fleksy Keyboard:
>
> Same as Google Keyboard
>
> Results for Multiling O Keyboard:
>
> - Compositionevents are triggered.
> - Keydown events behave similar, but there are more exceptions where it
> behaves like it should: backspace within a word is inside a composition,
> while backspace between words or from one line to another triggers a
> properkeydown event that can be canceled.
>
- The user can still merge paragraphs in a way that cannot be canceled:
> mark content from line 1 to line 2, hit any letter on the keyboard.
>
>
> Results for Hacker's Keyboard:
>
> - Does not trigger composition events.
> - Does trigegr proper keydown event for backspace, even within a word
>

IME apps on Android have the option at their leisure of sending fake
physical key events instead of the IME-style InputConnection calls.
Hacker's Keyboard is sending these fake events in all cases, and Multiling
O is sending them just for backspaces (as AOSP keyboard once did or perhaps
still does).  Chromium has no way of telling such events apart from a
physical keyboard literally plugged into the phone (which is possible with
Bluetooth or USB-C hosting), so we treat them as physical key events,
cancelability and all.


>
> ---
>
> *Looking at Code:*
>
> The Hacker's keyboard is open source so we can look at that.
>

I'd suggest especially looking at AOSP's Keyboard source -- it's an early
ancestor of almost every keyboard app.  Even though they've diverged
significantly from it over time, AOSP Keyboard is still a useful reference
implementation.  It's canonically hosted here
<https://android.googlesource.com/platform/packages/inputmethods/LatinIME/+/master>
and
the file especially of interest is RichInputConnection.java .

The IME code seems fairly easy to understand and I was looking for the part
> that handles backspace in the Latin evrsion. I found it here [1]. From the
> looks of it, the reason I received the proper keydown event was that in
> many standard cases, it just sends a sendDownUpKeyEvents() command with the
> given keycode.
>
>
> ---
>
> *Looking at documentation [1]:*
>
> There only seem to be a limited number of ways the IME can read and write
> to the document.
>
> For reading:
> - It can extract text of the selection and text before and after it
> - it can get style information about parts before and after it, but as far
> as I can tell no information about soft line breaks (where does the
> breaking actually take palce on the screen)
>
- It can subscribe to updates to the text it is working on
>
>
> For writing:
> - It can send key events
> - It can make a change to the text in the composition, but it doesn't have
> a way of explaining the semantic meaning of that change
> - It can delete X characters before or after the selection
> - It can perform a predefined editor action (none of which seem to be used
> by the Hacker's keyboard)
> - It can cut or paste
>

Agreed with this.  I would summarize this by saying that IMEs operate on a
"plaintext distillation" of the text, which is 1-to-1 with the data
structure of <textarea>, but is the homomorphic image of a contenteditable.


>
> There are also ways of setting what the composing text is and finishign
> the composition. But I saw that used little in the Hacker's Keyboard source
> code, and I assume that the common practice among keyboard makers is to
> ignore the methods related to the composition string and this is then the
> reason why there are no composition-related events being triggered.
>
>
> ---
>
> If this is all correct, it means that a keyboard may implement a way to
> delete a word wgoing forward, but it won't be able to tell the browser that
> that is the semantic meaning of what it is trying to do. The browser just
> knows that the user has requested to delete for example 15 characters
> forward. So it seems like those events that deal with such semantic type
> editing are not relevant at all for Android input currently. If you do
> decide to make such commands in the future, then I don't see why you would
> have to wrap them in IEM either -- it would create many fewer problems if
> such edits would come outside of it, like the Enter-key event.
>

Yes, I agree.  The only (small and tractable) problem is that today,
especially in contenteditable the distinction between semantic meaning
operations or not is slightly ambiguous, so I wasn't sure if there was a
preexisting case where commitText("\n\n") might be mapped to
insertParagraph, for instance.  I think it could be hashed out and
clarified, it's just that we had to quickly make a decision on which ones
to exclude.


> Some of these things may really depend on the keyboard, but if the Google
> keyboard would work well in contenteditable, then I'm sure competitors
> would also make sure that their keyboard would start working correctly.
>

This is a niche consideration from IME apps' point of view, given the
dearth of already existing Android Javascript rich editors.  So it's best
to assume they feel absolutely zero competitive pressure around this, and
any change we might reach standards consensus on making can only be made in
Chromium's source code.

It would be interesting to hear why it was chosen not to trigger
> composition events.
>

>
As for backspace: It would be interesting to hear why it was chosen to wrap
> this in IME, even between words or between paragraphs.
>

I think of an Android IME as having its own MVC architecture with a mirror
image of DOM diffing.  They have their own internal data structure for the
textbox as well as a possibly stale idea of what the editor process (that's
us) has in its textbox structure.  When user presses soft backspace (or any
other soft key), they first apply the change to their model, and then run a
diffing algorithm against the previous model to generate IME commands to
send to the editor process for reconciliation.  For backspace, this command
is typically deleteSurroundingText(0, 1), but depending on the vagaries of
the diffing algorithm, it's sometimes much more complex.  (Note that the
AOSP keyboard source doesn't appear to illustrate this, unfortunately.)

This does make things a lot more complicated for JS editors as they will
> have to wait for the end of the composition and then potentially undo
> everything that was doen during the composition. So we are back at diffing
> the DOM.
>

> There is one case that we cannot really seem to address as long as IME is
> the black box it is: If a user selects text from paragraph 1 to 3 and then
> hits a signle letter on the keyboard, the browser just merges those
> paragraphs. This is of coruse resolved in level 2 of the spec, but given
> that you don't want to implement that level, I guess the only other way is
> to giving more control over the IME to JS itself.
>

>
>
>
>
>
>
>
>
>
>
> [1] https://github.com/klausw/hackerskeyboard/blob/
> 3cefa2f89797d5c55d49b900d8caed7074582ef0/java/src/org/
> pocketworkstation/pckeyboard/LatinIME.java#L2137
>
>
> [2] https://developer.android.com/reference/android/view/
> inputmethod/InputConnection.html
>
>
>
>
>
>
>
>
> On Tue, Jun 27, 2017 at 2:04 AM, Johannes Wilm <mail@johanneswilm.org>
> wrote:
>
>> Hey Alexandre,
>> as far as our agreement hitherto had been, only a few of these will ever
>> be triggered during IME, and those are exclusive to IME. So in other words:
>> those input types that are used in IME are not used outside of it and vice
>> versa. I understand that on Android, a lot of things have been wrapped in
>> IME, which seems like a different problem that we hopefully will resolve
>> once we get more control over the IME process.
>>
>> In effect it would mean that on Android, most of the inputTypes would
>> likely not be in use. For example -- on some desktop OSes there are special
>> keyboard shortcuts for operations such as deleting from the caret position
>> to the next softline wrapping. On other OSes without such a keyboard
>> shortcut, these inputTypes would not be triggered at all.
>>
>> > because "Paragraph break" and "Delete word backward" are plain-text
>> operations
>>
>> Not necessarily. If a word is partially styled, then deleting that word
>> may mean several other DOM changes. SImilarly, if the caret is in an
>> italic-styled part of a line that is not fully styled, deleting to the end
>> of the line will not just be a plaintext operation.
>>
>>
>> The examples I mentioned were just randomly chosen. We'll need to build a
>> strong argument fro JS devs to be using the beforeinput event, and the part
>> about why some things can be canceled while others cannot still seems a bit
>> weak.
>>
>> On Tue, Jun 27, 2017 at 12:40 AM, Alexandre Elias <aelias@chromium.org>
>> wrote:
>>
>>> The philosophy is indeed that an IME might send it, and we erred a bit
>>> on the side of conservatism on the list to avoid locking ourselves in a bad
>>> situation if an IME does wind up sending it.  IMEs do not understand rich
>>> text nor the clipboard so it was safe to exclude ordered lists and "cut".
>>> But they have a wide variety of APIs to alter plain text, and because
>>> "Paragraph break" and "Delete word backward" are plain-text operations (in
>>> contenteditable, simple plain-text carriage returns might be mapped to
>>> <p>), it's possible one of them might send it.
>>>
>>> I'm not totally opposed to making these 2 cancelable if there is a clear
>>> use case and we can guarantee that we never map IME operations to it but to
>>> an alternate event, though.  Or if the problem is mostly confusion rather
>>> than a use case, we can explain the reason is "plain textiness ~= IME".
>>>
>>>
>>> On Mon, Jun 26, 2017 at 2:17 PM, Rick Byers <rbyers@chromium.org> wrote:
>>>
>>>> +aelias.  Personally I really hope we can achieve interop around all
>>>> this :-)
>>>>
>>>> On Mon, Jun 26, 2017 at 4:43 PM, Johannes Wilm <mail@johanneswilm.org>
>>>> wrote:
>>>>
>>>>> Hey,
>>>>> the TAG review of the input events spec asked us among other things to
>>>>> convert the long list of inputType values to a table. I have now done that
>>>>> for both levels. Now that I am done I get a clearer picture of which iof
>>>>> the inputTypes the Google people decided to not make cancelable in the
>>>>> level 1 version. They are marked as "Undefined" so that also those
>>>>> implementing level 2 are compliant with level 1. [1]
>>>>>
>>>>> I wonder though -- what made you guys decide to make exactly those not
>>>>> be cancelable? For example, inserting an ordered list can be canceled, but
>>>>> inserting a paragraph break cannot be canceled. Neither one of these should
>>>>> be happening during an IME composition, right? Or if I hit a special key
>>>>> combination to delete a word backward, that cannot be canceled. But if
>>>>> instead I select the word and hit the key combination for cut, that is
>>>>> cancelable.
>>>>>
>>>>> Is this list meant to be final or is this still work in progress? If
>>>>> this is final, then maybe it would be a good idea to write up soem
>>>>> explanation fro the philosophy that lies behind these choices. Otherwise I
>>>>> am sure a lot of JS devs will be just as confused as me about this.
>>>>>
>>>>>
>>>>> [1] https://rawgit.com/w3c/input-events/v1/index.html#h-inte
>>>>> rface-inputevent-attributes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Johannes Wilm
>>>>> http://www.johanneswilm.org
>>>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Johannes Wilm
>> http://www.johanneswilm.org
>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>
>
>
>
> --
> Johannes Wilm
> http://www.johanneswilm.org
> tel: +1 (520) 399 8880 <(520)%20399-8880>
>