Re: cancelability in level 1 of input-events spec

Hey again,

I've been trying to understand the problem you guys have on Android a bit
better and have done some testing with different keyboards on Android and
reading up on what I could find of documentation of IME on Android.

This mroning I consulted with Piotr Koszuliński from CKEditor and had
earlier discussed with Marijn Haverbeke from ProseMirror specifically about
contenteditable issues on Android. Then I did some testing myself.

I may have misunderstood some things, so please enlighten me in case I
should be wrong.

In brief, it seems like:

A. On Android, the right keyboard related events are sent only if the
keyboard has implemented everything correctly.
B. The standard, Google-provided keyboard, is not sending the rights
events, and those events that are sent have incorrect information.
C. There may be some things that are defined by the browser about these
events that is giving incorrect information to JS, given that all keyboard
tested showed some common bugs.
D. The Android IME interface allows the IME to edit the document in a few
limited ways, none of which include the semantic editing operations which
you have decided to make non-cancelable in Level 1 of the Input Events spec.

----

*Testing: *

I have looked at the Google Keyboard (Gboard), Fleksy Keyboard, Multiling O
Keyboard and Hacker's Keyboard on contenteditable and operations:

typing character within word
backspace within word
backspace between words
backspace at beginning of paragraph (with another previous paragraph)
selecting text from one paragraph to another and then typing a character


Result for Google Keyboard:

- On Android, no composition events (compositionstart, compositionupdate,
compositionend) are triggered, no matter what. This looks like a bug.
- The keydown event sometimes has isComposing = true, and sometimes
isComposing = false, but this seems to have no influence over how it works.
This looks like a bug.
- The keydown events on Android claim they are cancelable, but
event.preventDefault() does nothing. This looks like a bug.
- The keydown events all have keyCode = 229, which is also what desktop
browsers do during IME.
- An exception is the Enter key, which does give a keyCode, and which can
be canceled with event.preventDefault().

Results for Fleksy Keyboard:

Same as Google Keyboard

Results for Multiling O Keyboard:

- Compositionevents are triggered.
- Keydown events behave similar, but there are more exceptions where it
behaves like it should: backspace within a word is inside a composition,
while backspace between words or from one line to another triggers a
properkeydown event that can be canceled.
- The user can still merge paragraphs in a way that cannot be canceled:
mark content from line 1 to line 2, hit any letter on the keyboard.


Results for Hacker's Keyboard:

- Does not trigger composition events.
- Does trigegr proper keydown event for backspace, even within a word


---

*Looking at Code:*

The Hacker's keyboard is open source so we can look at that. The IME code
seems fairly easy to understand and I was looking for the part that handles
backspace in the Latin evrsion. I found it here [1]. From the looks of it,
the reason I received the proper keydown event was that in many standard
cases, it just sends a sendDownUpKeyEvents() command with the given keycode.


---

*Looking at documentation [1]:*

There only seem to be a limited number of ways the IME can read and write
to the document.

For reading:
- It can extract text of the selection and text before and after it
- it can get style information about parts before and after it, but as far
as I can tell no information about soft line breaks (where does the
breaking actually take palce on the screen)
- It can subscribe to updates to the text it is working on


For writing:
- It can send key events
- It can make a change to the text in the composition, but it doesn't have
a way of explaining the semantic meaning of that change
- It can delete X characters before or after the selection
- It can perform a predefined editor action (none of which seem to be used
by the Hacker's keyboard)
- It can cut or paste


There are also ways of setting what the composing text is and finishign the
composition. But I saw that used little in the Hacker's Keyboard source
code, and I assume that the common practice among keyboard makers is to
ignore the methods related to the composition string and this is then the
reason why there are no composition-related events being triggered.


---

If this is all correct, it means that a keyboard may implement a way to
delete a word wgoing forward, but it won't be able to tell the browser that
that is the semantic meaning of what it is trying to do. The browser just
knows that the user has requested to delete for example 15 characters
forward. So it seems like those events that deal with such semantic type
editing are not relevant at all for Android input currently. If you do
decide to make such commands in the future, then I don't see why you would
have to wrap them in IEM either -- it would create many fewer problems if
such edits would come outside of it, like the Enter-key event.

Some of these things may really depend on the keyboard, but if the Google
keyboard would work well in contenteditable, then I'm sure competitors
would also make sure that their keyboard would start working correctly. It
would be interesting to hear why it was chosen not to trigger composition
events.

As for backspace: It would be interesting to hear why it was chosen to wrap
this in IME, even between words or between paragraphs. This does make
things a lot more complicated for JS editors as they will have to wait for
the end of the composition and then potentially undo everything that was
doen during the composition. So we are back at diffing the DOM.

There is one case that we cannot really seem to address as long as IME is
the black box it is: If a user selects text from paragraph 1 to 3 and then
hits a signle letter on the keyboard, the browser just merges those
paragraphs. This is of coruse resolved in level 2 of the spec, but given
that you don't want to implement that level, I guess the only other way is
to giving more control over the IME to JS itself.











[1]
https://github.com/klausw/hackerskeyboard/blob/3cefa2f89797d5c55d49b900d8caed7074582ef0/java/src/org/pocketworkstation/pckeyboard/LatinIME.java#L2137


[2]
https://developer.android.com/reference/android/view/inputmethod/InputConnection.html








On Tue, Jun 27, 2017 at 2:04 AM, Johannes Wilm <mail@johanneswilm.org>
wrote:

> Hey Alexandre,
> as far as our agreement hitherto had been, only a few of these will ever
> be triggered during IME, and those are exclusive to IME. So in other words:
> those input types that are used in IME are not used outside of it and vice
> versa. I understand that on Android, a lot of things have been wrapped in
> IME, which seems like a different problem that we hopefully will resolve
> once we get more control over the IME process.
>
> In effect it would mean that on Android, most of the inputTypes would
> likely not be in use. For example -- on some desktop OSes there are special
> keyboard shortcuts for operations such as deleting from the caret position
> to the next softline wrapping. On other OSes without such a keyboard
> shortcut, these inputTypes would not be triggered at all.
>
> > because "Paragraph break" and "Delete word backward" are plain-text
> operations
>
> Not necessarily. If a word is partially styled, then deleting that word
> may mean several other DOM changes. SImilarly, if the caret is in an
> italic-styled part of a line that is not fully styled, deleting to the end
> of the line will not just be a plaintext operation.
>
>
> The examples I mentioned were just randomly chosen. We'll need to build a
> strong argument fro JS devs to be using the beforeinput event, and the part
> about why some things can be canceled while others cannot still seems a bit
> weak.
>
> On Tue, Jun 27, 2017 at 12:40 AM, Alexandre Elias <aelias@chromium.org>
> wrote:
>
>> The philosophy is indeed that an IME might send it, and we erred a bit on
>> the side of conservatism on the list to avoid locking ourselves in a bad
>> situation if an IME does wind up sending it.  IMEs do not understand rich
>> text nor the clipboard so it was safe to exclude ordered lists and "cut".
>> But they have a wide variety of APIs to alter plain text, and because
>> "Paragraph break" and "Delete word backward" are plain-text operations (in
>> contenteditable, simple plain-text carriage returns might be mapped to
>> <p>), it's possible one of them might send it.
>>
>> I'm not totally opposed to making these 2 cancelable if there is a clear
>> use case and we can guarantee that we never map IME operations to it but to
>> an alternate event, though.  Or if the problem is mostly confusion rather
>> than a use case, we can explain the reason is "plain textiness ~= IME".
>>
>>
>> On Mon, Jun 26, 2017 at 2:17 PM, Rick Byers <rbyers@chromium.org> wrote:
>>
>>> +aelias.  Personally I really hope we can achieve interop around all
>>> this :-)
>>>
>>> On Mon, Jun 26, 2017 at 4:43 PM, Johannes Wilm <mail@johanneswilm.org>
>>> wrote:
>>>
>>>> Hey,
>>>> the TAG review of the input events spec asked us among other things to
>>>> convert the long list of inputType values to a table. I have now done that
>>>> for both levels. Now that I am done I get a clearer picture of which iof
>>>> the inputTypes the Google people decided to not make cancelable in the
>>>> level 1 version. They are marked as "Undefined" so that also those
>>>> implementing level 2 are compliant with level 1. [1]
>>>>
>>>> I wonder though -- what made you guys decide to make exactly those not
>>>> be cancelable? For example, inserting an ordered list can be canceled, but
>>>> inserting a paragraph break cannot be canceled. Neither one of these should
>>>> be happening during an IME composition, right? Or if I hit a special key
>>>> combination to delete a word backward, that cannot be canceled. But if
>>>> instead I select the word and hit the key combination for cut, that is
>>>> cancelable.
>>>>
>>>> Is this list meant to be final or is this still work in progress? If
>>>> this is final, then maybe it would be a good idea to write up soem
>>>> explanation fro the philosophy that lies behind these choices. Otherwise I
>>>> am sure a lot of JS devs will be just as confused as me about this.
>>>>
>>>>
>>>> [1] https://rawgit.com/w3c/input-events/v1/index.html#h-inte
>>>> rface-inputevent-attributes
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Johannes Wilm
>>>> http://www.johanneswilm.org
>>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>>
>>>
>>>
>>
>
>
> --
> Johannes Wilm
> http://www.johanneswilm.org
> tel: +1 (520) 399 8880 <(520)%20399-8880>
>



-- 
Johannes Wilm
http://www.johanneswilm.org
tel: +1 (520) 399 8880