Re: cancelability in level 1 of input-events spec

Hey Alexandre,


On Wed, Jun 28, 2017 at 3:33 AM, Alexandre Elias <aelias@chromium.org>
wrote:

> Thanks for taking the time to better understand Android behavior.
>
> On Tue, Jun 27, 2017 at 8:19 AM, Johannes Wilm <mail@johanneswilm.org>
> wrote:
>
>> Hey again,
>>
>> I've been trying to understand the problem you guys have on Android a bit
>> better and have done some testing with different keyboards on Android and
>> reading up on what I could find of documentation of IME on Android.
>>
>> This mroning I consulted with Piotr Koszuliński from CKEditor and had
>> earlier discussed with Marijn Haverbeke from ProseMirror specifically about
>> contenteditable issues on Android. Then I did some testing myself.
>>
>> I may have misunderstood some things, so please enlighten me in case I
>> should be wrong.
>>
>> In brief, it seems like:
>>
>> A. On Android, the right keyboard related events are sent only if the
>> keyboard has implemented everything correctly.
>> B. The standard, Google-provided keyboard, is not sending the rights
>> events, and those events that are sent have incorrect information.
>>
>
> From Chromium-team's point of view, any pattern of Java API calls that
> Google Keyboard (or for that matter Samsung Keyboard, LG keyboard etc) does
> is by definition "right", and something we have very little power to
> change.  Chromium-team has no power to dictate to Android keyboard apps
> whatever behavior we might subjectively think is "right".
>

Right. I should have phrased this differently. With "right" I meant
"following relevant specifications". When JS receives no composition events
even thouhg an IME composition takes place, that does not seem right. I
don't know ifn this is something you could enforce in the chromium code
("If no compositionstart event has been sent at the time an IME-based
non-cancelable  keydown event comes, send a compositionstart event first,
...") or if there are other things preventing this. But if the Multiling O
Keyboard is able to send these events, it would seem that Google's
keyboards also should be programable in a way that is spec-compliant.



>
> However, in those cases where a problem is 100% due to how Chromium
> interprets/mangles the Google Keyboard Java API calls to JS, I *can* do
> something about it.
>
> C. There may be some things that are defined by the browser about these
>> events that is giving incorrect information to JS, given that all keyboard
>> tested showed some common bugs.
>> D. The Android IME interface allows the IME to edit the document in a few
>> limited ways, none of which include the semantic editing operations which
>> you have decided to make non-cancelable in Level 1 of the Input Events spec.
>>
>> ----
>>
>> *Testing: *
>>
>> I have looked at the Google Keyboard (Gboard), Fleksy Keyboard, Multiling
>> O Keyboard and Hacker's Keyboard on contenteditable and operations:
>>
>> typing character within word
>> backspace within word
>> backspace between words
>> backspace at beginning of paragraph (with another previous paragraph)
>> selecting text from one paragraph to another and then typing a character
>>
>>
>> Result for Google Keyboard:
>>
>> - On Android, no composition events (compositionstart, compositionupdate,
>> compositionend) are triggered, no matter what. This looks like a bug.
>>
>
> I would indeed consider that a bug, but I cannot reproduce this
> observation.  On this test page
> <https://cdn.rawgit.com/w3c/uievents/gh-pages/tools/key-event-viewer.html>,
> using Chrome Beta 60.0.3112.43 and GBoard Beta 6.3.30.159940404 on Galaxy
> S7/NRD90M, I see the compositionStart/Update/End events just fine.  Let me
> know what exact versions of all those things you observed this with.
>

I was using Chrome 58 with gboard 6.3.30.xxx (both latest stable versions),
but I have now switched to the beta version of Chromium. I have also signed
up for the gboard beta program, but it will take some time to be fully
processed into that program, I understand.

Right now there are still no composition events, but I notice that there
are now beforeinput events, but they use the inputTypes meant for non-IME
input (inputText and deleteContentBackward) instead of the IME one
(insertCompositionText).

So I wonder - either my Android keyboard is not in a composition, in which
case all the keydown/beforeinput events should be cancelable and the the
keycodes be correct, or we are in a composition, in which case the
composition events are missing and the beforeinput input types are
incorrect.


>
>> - The keydown event sometimes has isComposing = true, and sometimes
>> isComposing = false, but this seems to have no influence over how it works.
>> This looks like a bug.
>>
>
> It seems to accurately reflect whether a composing underline is currently
> present.  I'm not sure that anything else is specified in terms of
> influencing how it works.  In what sense is the current havior a bug?
>

If no composition events have been sent, then either:

- This is part of a composition, in which case the isComposition is correct
when true and incorrect when false
- This is not part of a composition, in which case isComposition should
always be false, and there shouldn't be any reason for the the keycodes of
the keydown events are all 229 and the event beign non-cancelable.


>
>
>> - The keydown events on Android claim they are cancelable, but
>> event.preventDefault() does nothing. This looks like a bug.
>>
>
> Thanks.  I agree that is a bug and one that we're fully empowered to fix
> within Chromium (at least to the extent of making cancelable accurately
> false).  I filed http://crbug.com/737349 to track it.
>
>
>> - The keydown events all have keyCode = 229, which is also what desktop
>> browsers do during IME.
>>
>
> Yes.  The choice of 229, a number originating from MS Windows, isn't ideal
> on Android because 229 is also mapped to LAST_CHANNEL (Android TV remote
> key).  We are considering changing it to 0 keycode instead, as some other
> desktop browsers do.
>
>
>> - An exception is the Enter key, which does give a keyCode, and which can
>> be canceled with event.preventDefault().
>>
>
> This is a known problem.  In Java InputConnection, Google Keyboard still
> generates the usual commitText("\n") call for Enter, but Chromium was
> forced to introduce a workaround for the Android WebView-based GMail native
> app, which relies on the keyCode for its special inline reply break
> behavior.  This is brittle and doesn't work properly in all cases, so we're
> working with them to transition to a better approach, after which we hope
> to remove this behavior.  Removal is tracked in http://crbug.com/579652
>


Based on the comments on that bug report and what I have heard from JS
devs, it would seem JS devs would actually prefer to have them stay being
real keys, both Enter and backspace, at least in those cases where they are
not used as part of a composition. I am not sure why you would want to
change this. Maybe the problem gets smaller once we get to cancel IME input
+ the beforeinput event rolled out though.




>
>>
>> Results for Fleksy Keyboard:
>>
>> Same as Google Keyboard
>>
>> Results for Multiling O Keyboard:
>>
>> - Compositionevents are triggered.
>> - Keydown events behave similar, but there are more exceptions where it
>> behaves like it should: backspace within a word is inside a composition,
>> while backspace between words or from one line to another triggers a
>> properkeydown event that can be canceled.
>>
> - The user can still merge paragraphs in a way that cannot be canceled:
>> mark content from line 1 to line 2, hit any letter on the keyboard.
>>
>>
>> Results for Hacker's Keyboard:
>>
>> - Does not trigger composition events.
>> - Does trigegr proper keydown event for backspace, even within a word
>>
>
> IME apps on Android have the option at their leisure of sending fake
> physical key events instead of the IME-style InputConnection calls.
> Hacker's Keyboard is sending these fake events in all cases, and Multiling
> O is sending them just for backspaces (as AOSP keyboard once did or perhaps
> still does).  Chromium has no way of telling such events apart from a
> physical keyboard literally plugged into the phone (which is possible with
> Bluetooth or USB-C hosting), so we treat them as physical key events,
> cancelability and all.
>


Yes, and actually such key down events are OK to process. That removes one
potentially problem for JS devs.

>
>
>>
>> ---
>>
>> *Looking at Code:*
>>
>> The Hacker's keyboard is open source so we can look at that.
>>
>
> I'd suggest especially looking at AOSP's Keyboard source -- it's an early
> ancestor of almost every keyboard app.  Even though they've diverged
> significantly from it over time, AOSP Keyboard is still a useful reference
> implementation.  It's canonically hosted here
> <https://android.googlesource.com/platform/packages/inputmethods/LatinIME/+/master> and
> the file especially of interest is RichInputConnection.java .
>

Thanks! And yes, I had noticed that Hacker's keyboard referred to the AOSP
keyboard.

>
> The IME code seems fairly easy to understand and I was looking for the
>> part that handles backspace in the Latin evrsion. I found it here [1]. From
>> the looks of it, the reason I received the proper keydown event was that in
>> many standard cases, it just sends a sendDownUpKeyEvents() command with the
>> given keycode.
>>
>>
>> ---
>>
>> *Looking at documentation [1]:*
>>
>> There only seem to be a limited number of ways the IME can read and write
>> to the document.
>>
>> For reading:
>> - It can extract text of the selection and text before and after it
>> - it can get style information about parts before and after it, but as
>> far as I can tell no information about soft line breaks (where does the
>> breaking actually take palce on the screen)
>>
> - It can subscribe to updates to the text it is working on
>>
>>
>> For writing:
>> - It can send key events
>> - It can make a change to the text in the composition, but it doesn't
>> have a way of explaining the semantic meaning of that change
>> - It can delete X characters before or after the selection
>> - It can perform a predefined editor action (none of which seem to be
>> used by the Hacker's keyboard)
>> - It can cut or paste
>>
>
> Agreed with this.  I would summarize this by saying that IMEs operate on a
> "plaintext distillation" of the text, which is 1-to-1 with the data
> structure of <textarea>, but is the homomorphic image of a contenteditable.
>

OK, just from looking at the documentation, it seemed like the keyboard can
get some style information as well (just read, not write), if it decides to
do so. Incorrect?


>
>
>>
>> There are also ways of setting what the composing text is and finishign
>> the composition. But I saw that used little in the Hacker's Keyboard source
>> code, and I assume that the common practice among keyboard makers is to
>> ignore the methods related to the composition string and this is then the
>> reason why there are no composition-related events being triggered.
>>
>>
>> ---
>>
>> If this is all correct, it means that a keyboard may implement a way to
>> delete a word wgoing forward, but it won't be able to tell the browser that
>> that is the semantic meaning of what it is trying to do. The browser just
>> knows that the user has requested to delete for example 15 characters
>> forward. So it seems like those events that deal with such semantic type
>> editing are not relevant at all for Android input currently. If you do
>> decide to make such commands in the future, then I don't see why you would
>> have to wrap them in IEM either -- it would create many fewer problems if
>> such edits would come outside of it, like the Enter-key event.
>>
>
> Yes, I agree.  The only (small and tractable) problem is that today,
> especially in contenteditable the distinction between semantic meaning
> operations or not is slightly ambiguous, so I wasn't sure if there was a
> preexisting case where commitText("\n\n") might be mapped to
> insertParagraph, for instance.  I think it could be hashed out and
> clarified, it's just that we had to quickly make a decision on which ones
> to exclude.
>


Ok, but sound slike currently you are mapping commitText("\n\n") to a
cancelable Enter-key event otuside of IME, for which then also the
beforeinput event could be cancelable, right?

Once you remove this again, it should instead fire a beforeinput event with
the inputType="insertCompositionText", which we already have defined as
being non-cancelable.

I don't see why you need to modify the cancelability of all these non-IME
inputTypes that either don't apply to Android, or if they apply it is
because Android uses them outside of an IME.



>
>> Some of these things may really depend on the keyboard, but if the Google
>> keyboard would work well in contenteditable, then I'm sure competitors
>> would also make sure that their keyboard would start working correctly.
>>
>
> This is a niche consideration from IME apps' point of view, given the
> dearth of already existing Android Javascript rich editors.  So it's best
> to assume they feel absolutely zero competitive pressure around this, and
> any change we might reach standards consensus on making can only be made in
> Chromium's source code.
>


Well, it is these JavaScript editors that are diffing the DOM in various
ways on Android who feel what they are doing is imperfectand a lot of the
effort behind the discussions here is to find a way to make their job
easier (and the text output they produce more perfect).


>
> It would be interesting to hear why it was chosen not to trigger
>> composition events.
>>
>
>>
> As for backspace: It would be interesting to hear why it was chosen to
>> wrap this in IME, even between words or between paragraphs.
>>
>
> I think of an Android IME as having its own MVC architecture with a mirror
> image of DOM diffing.  They have their own internal data structure for the
> textbox as well as a possibly stale idea of what the editor process (that's
> us) has in its textbox structure.  When user presses soft backspace (or any
> other soft key), they first apply the change to their model, and then run a
> diffing algorithm against the previous model to generate IME commands to
> send to the editor process for reconciliation.  For backspace, this command
> is typically deleteSurroundingText(0, 1), but depending on the vagaries of
> the diffing algorithm, it's sometimes much more complex.  (Note that the
> AOSP keyboard source doesn't appear to illustrate this, unfortunately.)
>

Right, but then the JavaScript editor takes over and mangles the DOM in
some different way than what the IME expects and then nothing really works.
As far as I understand,. you guys actually tell the keyboard producers that
they need to be aware of DOm changes that they haven't introduced
themselves and that they should subscribe to updates to the text.

It seems IMEs are mainly a headache for JS devs because they cannot be
canceled and until now it was also difficult to know what the user tried to
do. If the editor needs to do its own DOM updates , for example because we
are in a collaborative editor, the problem is getting worse if the IEM is
just continous with no interruptable events and never reading back the
actual DOM. So we should keep that to a minimum.


>
> This does make things a lot more complicated for JS editors as they will
>> have to wait for the end of the composition and then potentially undo
>> everything that was doen during the composition. So we are back at diffing
>> the DOM.
>>
>
>> There is one case that we cannot really seem to address as long as IME is
>> the black box it is: If a user selects text from paragraph 1 to 3 and then
>> hits a signle letter on the keyboard, the browser just merges those
>> paragraphs. This is of coruse resolved in level 2 of the spec, but given
>> that you don't want to implement that level, I guess the only other way is
>> to giving more control over the IME to JS itself.
>>
>
>>


If the problem here was only a misunderstanding of the usage of the
inputTypes, and Chromium can be changed to be spec compliant in the choice
of input type (level 1 compliance), then there really isn't any need to
keep the difference in cancelability for all these other input types. The
difference between level 1 and level 2 of the spec would then simply boil
down to level 2 having 4 IME-related inputTypes and level 1 having just 1.




>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [1] https://github.com/klausw/hackerskeyboard/blob/3cefa2f89
>> 797d5c55d49b900d8caed7074582ef0/java/src/org/pocketworkstati
>> on/pckeyboard/LatinIME.java#L2137
>>
>>
>> [2] https://developer.android.com/reference/android/view/inp
>> utmethod/InputConnection.html
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jun 27, 2017 at 2:04 AM, Johannes Wilm <mail@johanneswilm.org>
>> wrote:
>>
>>> Hey Alexandre,
>>> as far as our agreement hitherto had been, only a few of these will ever
>>> be triggered during IME, and those are exclusive to IME. So in other words:
>>> those input types that are used in IME are not used outside of it and vice
>>> versa. I understand that on Android, a lot of things have been wrapped in
>>> IME, which seems like a different problem that we hopefully will resolve
>>> once we get more control over the IME process.
>>>
>>> In effect it would mean that on Android, most of the inputTypes would
>>> likely not be in use. For example -- on some desktop OSes there are special
>>> keyboard shortcuts for operations such as deleting from the caret position
>>> to the next softline wrapping. On other OSes without such a keyboard
>>> shortcut, these inputTypes would not be triggered at all.
>>>
>>> > because "Paragraph break" and "Delete word backward" are plain-text
>>> operations
>>>
>>> Not necessarily. If a word is partially styled, then deleting that word
>>> may mean several other DOM changes. SImilarly, if the caret is in an
>>> italic-styled part of a line that is not fully styled, deleting to the end
>>> of the line will not just be a plaintext operation.
>>>
>>>
>>> The examples I mentioned were just randomly chosen. We'll need to build
>>> a strong argument fro JS devs to be using the beforeinput event, and the
>>> part about why some things can be canceled while others cannot still seems
>>> a bit weak.
>>>
>>> On Tue, Jun 27, 2017 at 12:40 AM, Alexandre Elias <aelias@chromium.org>
>>> wrote:
>>>
>>>> The philosophy is indeed that an IME might send it, and we erred a bit
>>>> on the side of conservatism on the list to avoid locking ourselves in a bad
>>>> situation if an IME does wind up sending it.  IMEs do not understand rich
>>>> text nor the clipboard so it was safe to exclude ordered lists and "cut".
>>>> But they have a wide variety of APIs to alter plain text, and because
>>>> "Paragraph break" and "Delete word backward" are plain-text operations (in
>>>> contenteditable, simple plain-text carriage returns might be mapped to
>>>> <p>), it's possible one of them might send it.
>>>>
>>>> I'm not totally opposed to making these 2 cancelable if there is a
>>>> clear use case and we can guarantee that we never map IME operations to it
>>>> but to an alternate event, though.  Or if the problem is mostly confusion
>>>> rather than a use case, we can explain the reason is "plain textiness ~=
>>>> IME".
>>>>
>>>>
>>>> On Mon, Jun 26, 2017 at 2:17 PM, Rick Byers <rbyers@chromium.org>
>>>> wrote:
>>>>
>>>>> +aelias.  Personally I really hope we can achieve interop around all
>>>>> this :-)
>>>>>
>>>>> On Mon, Jun 26, 2017 at 4:43 PM, Johannes Wilm <mail@johanneswilm.org>
>>>>> wrote:
>>>>>
>>>>>> Hey,
>>>>>> the TAG review of the input events spec asked us among other things
>>>>>> to convert the long list of inputType values to a table. I have now done
>>>>>> that for both levels. Now that I am done I get a clearer picture of which
>>>>>> iof the inputTypes the Google people decided to not make cancelable in the
>>>>>> level 1 version. They are marked as "Undefined" so that also those
>>>>>> implementing level 2 are compliant with level 1. [1]
>>>>>>
>>>>>> I wonder though -- what made you guys decide to make exactly those
>>>>>> not be cancelable? For example, inserting an ordered list can be canceled,
>>>>>> but inserting a paragraph break cannot be canceled. Neither one of these
>>>>>> should be happening during an IME composition, right? Or if I hit a special
>>>>>> key combination to delete a word backward, that cannot be canceled. But if
>>>>>> instead I select the word and hit the key combination for cut, that is
>>>>>> cancelable.
>>>>>>
>>>>>> Is this list meant to be final or is this still work in progress? If
>>>>>> this is final, then maybe it would be a good idea to write up soem
>>>>>> explanation fro the philosophy that lies behind these choices. Otherwise I
>>>>>> am sure a lot of JS devs will be just as confused as me about this.
>>>>>>
>>>>>>
>>>>>> [1] https://rawgit.com/w3c/input-events/v1/index.html#h-inte
>>>>>> rface-inputevent-attributes
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Johannes Wilm
>>>>>> http://www.johanneswilm.org
>>>>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Johannes Wilm
>>> http://www.johanneswilm.org
>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>
>>
>>
>>
>> --
>> Johannes Wilm
>> http://www.johanneswilm.org
>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>
>
>


-- 
Johannes Wilm
http://www.johanneswilm.org
tel: +1 (520) 399 8880