Re: cancelability in level 1 of input-events spec

>
> I would indeed consider that a bug, but I cannot reproduce this
> observation.  On this test page
> <https://cdn.rawgit.com/w3c/uievents/gh-pages/tools/key-event-viewer.html>,
> using Chrome Beta 60.0.3112.43 and GBoard Beta 6.3.30.159940404 on Galaxy
> S7/NRD90M, I see the compositionStart/Update/End events just fine.  Let me
> know what exact versions of all those things you observed this with.
>


As a side note: I also checked the same on Safari 10.1. Safari 10.1 on Mac
OS X seems to be following level 2 of the spec correctly - both with IME
and traditional keyboard input.




On Wed, Jun 28, 2017 at 10:39 AM, Johannes Wilm <mail@johanneswilm.org>
wrote:

> Hey Alexandre,
>
>
> On Wed, Jun 28, 2017 at 3:33 AM, Alexandre Elias <aelias@chromium.org>
> wrote:
>
>> Thanks for taking the time to better understand Android behavior.
>>
>> On Tue, Jun 27, 2017 at 8:19 AM, Johannes Wilm <mail@johanneswilm.org>
>> wrote:
>>
>>> Hey again,
>>>
>>> I've been trying to understand the problem you guys have on Android a
>>> bit better and have done some testing with different keyboards on Android
>>> and reading up on what I could find of documentation of IME on Android.
>>>
>>> This mroning I consulted with Piotr Koszuliński from CKEditor and had
>>> earlier discussed with Marijn Haverbeke from ProseMirror specifically about
>>> contenteditable issues on Android. Then I did some testing myself.
>>>
>>> I may have misunderstood some things, so please enlighten me in case I
>>> should be wrong.
>>>
>>> In brief, it seems like:
>>>
>>> A. On Android, the right keyboard related events are sent only if the
>>> keyboard has implemented everything correctly.
>>> B. The standard, Google-provided keyboard, is not sending the rights
>>> events, and those events that are sent have incorrect information.
>>>
>>
>> From Chromium-team's point of view, any pattern of Java API calls that
>> Google Keyboard (or for that matter Samsung Keyboard, LG keyboard etc) does
>> is by definition "right", and something we have very little power to
>> change.  Chromium-team has no power to dictate to Android keyboard apps
>> whatever behavior we might subjectively think is "right".
>>
>
> Right. I should have phrased this differently. With "right" I meant
> "following relevant specifications". When JS receives no composition events
> even thouhg an IME composition takes place, that does not seem right. I
> don't know ifn this is something you could enforce in the chromium code
> ("If no compositionstart event has been sent at the time an IME-based
> non-cancelable  keydown event comes, send a compositionstart event first,
> ...") or if there are other things preventing this. But if the Multiling O
> Keyboard is able to send these events, it would seem that Google's
> keyboards also should be programable in a way that is spec-compliant.
>
>
>
>>
>> However, in those cases where a problem is 100% due to how Chromium
>> interprets/mangles the Google Keyboard Java API calls to JS, I *can* do
>> something about it.
>>
>> C. There may be some things that are defined by the browser about these
>>> events that is giving incorrect information to JS, given that all keyboard
>>> tested showed some common bugs.
>>> D. The Android IME interface allows the IME to edit the document in a
>>> few limited ways, none of which include the semantic editing operations
>>> which you have decided to make non-cancelable in Level 1 of the Input
>>> Events spec.
>>>
>>> ----
>>>
>>> *Testing: *
>>>
>>> I have looked at the Google Keyboard (Gboard), Fleksy Keyboard,
>>> Multiling O Keyboard and Hacker's Keyboard on contenteditable and
>>> operations:
>>>
>>> typing character within word
>>> backspace within word
>>> backspace between words
>>> backspace at beginning of paragraph (with another previous paragraph)
>>> selecting text from one paragraph to another and then typing a character
>>>
>>>
>>> Result for Google Keyboard:
>>>
>>> - On Android, no composition events (compositionstart,
>>> compositionupdate, compositionend) are triggered, no matter what. This
>>> looks like a bug.
>>>
>>
>>
>> I was using Chrome 58 with gboard 6.3.30.xxx (both latest stable
> versions), but I have now switched to the beta version of Chromium. I have
> also signed up for the gboard beta program, but it will take some time to
> be fully processed into that program, I understand.
>
> Right now there are still no composition events, but I notice that there
> are now beforeinput events, but they use the inputTypes meant for non-IME
> input (inputText and deleteContentBackward) instead of the IME one
> (insertCompositionText).
>
> So I wonder - either my Android keyboard is not in a composition, in which
> case all the keydown/beforeinput events should be cancelable and the the
> keycodes be correct, or we are in a composition, in which case the
> composition events are missing and the beforeinput input types are
> incorrect.
>
>
>>
>>> - The keydown event sometimes has isComposing = true, and sometimes
>>> isComposing = false, but this seems to have no influence over how it works.
>>> This looks like a bug.
>>>
>>
>> It seems to accurately reflect whether a composing underline is currently
>> present.  I'm not sure that anything else is specified in terms of
>> influencing how it works.  In what sense is the current havior a bug?
>>
>
> If no composition events have been sent, then either:
>
> - This is part of a composition, in which case the isComposition is
> correct when true and incorrect when false
> - This is not part of a composition, in which case isComposition should
> always be false, and there shouldn't be any reason for the the keycodes of
> the keydown events are all 229 and the event beign non-cancelable.
>
>
>>
>>
>>> - The keydown events on Android claim they are cancelable, but
>>> event.preventDefault() does nothing. This looks like a bug.
>>>
>>
>> Thanks.  I agree that is a bug and one that we're fully empowered to fix
>> within Chromium (at least to the extent of making cancelable accurately
>> false).  I filed http://crbug.com/737349 to track it.
>>
>>
>>> - The keydown events all have keyCode = 229, which is also what desktop
>>> browsers do during IME.
>>>
>>
>> Yes.  The choice of 229, a number originating from MS Windows, isn't
>> ideal on Android because 229 is also mapped to LAST_CHANNEL (Android TV
>> remote key).  We are considering changing it to 0 keycode instead, as some
>> other desktop browsers do.
>>
>>
>>> - An exception is the Enter key, which does give a keyCode, and which
>>> can be canceled with event.preventDefault().
>>>
>>
>> This is a known problem.  In Java InputConnection, Google Keyboard still
>> generates the usual commitText("\n") call for Enter, but Chromium was
>> forced to introduce a workaround for the Android WebView-based GMail native
>> app, which relies on the keyCode for its special inline reply break
>> behavior.  This is brittle and doesn't work properly in all cases, so we're
>> working with them to transition to a better approach, after which we hope
>> to remove this behavior.  Removal is tracked in http://crbug.com/579652
>>
>
>
> Based on the comments on that bug report and what I have heard from JS
> devs, it would seem JS devs would actually prefer to have them stay being
> real keys, both Enter and backspace, at least in those cases where they are
> not used as part of a composition. I am not sure why you would want to
> change this. Maybe the problem gets smaller once we get to cancel IME input
> + the beforeinput event rolled out though.
>
>
>
>
>>
>>>
>>> Results for Fleksy Keyboard:
>>>
>>> Same as Google Keyboard
>>>
>>> Results for Multiling O Keyboard:
>>>
>>> - Compositionevents are triggered.
>>> - Keydown events behave similar, but there are more exceptions where it
>>> behaves like it should: backspace within a word is inside a composition,
>>> while backspace between words or from one line to another triggers a
>>> properkeydown event that can be canceled.
>>>
>> - The user can still merge paragraphs in a way that cannot be canceled:
>>> mark content from line 1 to line 2, hit any letter on the keyboard.
>>>
>>>
>>> Results for Hacker's Keyboard:
>>>
>>> - Does not trigger composition events.
>>> - Does trigegr proper keydown event for backspace, even within a word
>>>
>>
>> IME apps on Android have the option at their leisure of sending fake
>> physical key events instead of the IME-style InputConnection calls.
>> Hacker's Keyboard is sending these fake events in all cases, and Multiling
>> O is sending them just for backspaces (as AOSP keyboard once did or perhaps
>> still does).  Chromium has no way of telling such events apart from a
>> physical keyboard literally plugged into the phone (which is possible with
>> Bluetooth or USB-C hosting), so we treat them as physical key events,
>> cancelability and all.
>>
>
>
> Yes, and actually such key down events are OK to process. That removes one
> potentially problem for JS devs.
>
>>
>>
>>>
>>> ---
>>>
>>> *Looking at Code:*
>>>
>>> The Hacker's keyboard is open source so we can look at that.
>>>
>>
>> I'd suggest especially looking at AOSP's Keyboard source -- it's an early
>> ancestor of almost every keyboard app.  Even though they've diverged
>> significantly from it over time, AOSP Keyboard is still a useful reference
>> implementation.  It's canonically hosted here
>> <https://android.googlesource.com/platform/packages/inputmethods/LatinIME/+/master> and
>> the file especially of interest is RichInputConnection.java .
>>
>
> Thanks! And yes, I had noticed that Hacker's keyboard referred to the AOSP
> keyboard.
>
>>
>> The IME code seems fairly easy to understand and I was looking for the
>>> part that handles backspace in the Latin evrsion. I found it here [1]. From
>>> the looks of it, the reason I received the proper keydown event was that in
>>> many standard cases, it just sends a sendDownUpKeyEvents() command with the
>>> given keycode.
>>>
>>>
>>> ---
>>>
>>> *Looking at documentation [1]:*
>>>
>>> There only seem to be a limited number of ways the IME can read and
>>> write to the document.
>>>
>>> For reading:
>>> - It can extract text of the selection and text before and after it
>>> - it can get style information about parts before and after it, but as
>>> far as I can tell no information about soft line breaks (where does the
>>> breaking actually take palce on the screen)
>>>
>> - It can subscribe to updates to the text it is working on
>>>
>>>
>>> For writing:
>>> - It can send key events
>>> - It can make a change to the text in the composition, but it doesn't
>>> have a way of explaining the semantic meaning of that change
>>> - It can delete X characters before or after the selection
>>> - It can perform a predefined editor action (none of which seem to be
>>> used by the Hacker's keyboard)
>>> - It can cut or paste
>>>
>>
>> Agreed with this.  I would summarize this by saying that IMEs operate on
>> a "plaintext distillation" of the text, which is 1-to-1 with the data
>> structure of <textarea>, but is the homomorphic image of a contenteditable.
>>
>
> OK, just from looking at the documentation, it seemed like the keyboard
> can get some style information as well (just read, not write), if it
> decides to do so. Incorrect?
>
>
>>
>>
>>>
>>> There are also ways of setting what the composing text is and finishign
>>> the composition. But I saw that used little in the Hacker's Keyboard source
>>> code, and I assume that the common practice among keyboard makers is to
>>> ignore the methods related to the composition string and this is then the
>>> reason why there are no composition-related events being triggered.
>>>
>>>
>>> ---
>>>
>>> If this is all correct, it means that a keyboard may implement a way to
>>> delete a word wgoing forward, but it won't be able to tell the browser that
>>> that is the semantic meaning of what it is trying to do. The browser just
>>> knows that the user has requested to delete for example 15 characters
>>> forward. So it seems like those events that deal with such semantic type
>>> editing are not relevant at all for Android input currently. If you do
>>> decide to make such commands in the future, then I don't see why you would
>>> have to wrap them in IEM either -- it would create many fewer problems if
>>> such edits would come outside of it, like the Enter-key event.
>>>
>>
>> Yes, I agree.  The only (small and tractable) problem is that today,
>> especially in contenteditable the distinction between semantic meaning
>> operations or not is slightly ambiguous, so I wasn't sure if there was a
>> preexisting case where commitText("\n\n") might be mapped to
>> insertParagraph, for instance.  I think it could be hashed out and
>> clarified, it's just that we had to quickly make a decision on which ones
>> to exclude.
>>
>
>
> Ok, but sound slike currently you are mapping commitText("\n\n") to a
> cancelable Enter-key event otuside of IME, for which then also the
> beforeinput event could be cancelable, right?
>
> Once you remove this again, it should instead fire a beforeinput event
> with the inputType="insertCompositionText", which we already have defined
> as being non-cancelable.
>
> I don't see why you need to modify the cancelability of all these non-IME
> inputTypes that either don't apply to Android, or if they apply it is
> because Android uses them outside of an IME.
>
>
>
>>
>>> Some of these things may really depend on the keyboard, but if the
>>> Google keyboard would work well in contenteditable, then I'm sure
>>> competitors would also make sure that their keyboard would start working
>>> correctly.
>>>
>>
>> This is a niche consideration from IME apps' point of view, given the
>> dearth of already existing Android Javascript rich editors.  So it's best
>> to assume they feel absolutely zero competitive pressure around this, and
>> any change we might reach standards consensus on making can only be made in
>> Chromium's source code.
>>
>
>
> Well, it is these JavaScript editors that are diffing the DOM in various
> ways on Android who feel what they are doing is imperfectand a lot of the
> effort behind the discussions here is to find a way to make their job
> easier (and the text output they produce more perfect).
>
>
>>
>> It would be interesting to hear why it was chosen not to trigger
>>> composition events.
>>>
>>
>>>
>> As for backspace: It would be interesting to hear why it was chosen to
>>> wrap this in IME, even between words or between paragraphs.
>>>
>>
>> I think of an Android IME as having its own MVC architecture with a
>> mirror image of DOM diffing.  They have their own internal data structure
>> for the textbox as well as a possibly stale idea of what the editor process
>> (that's us) has in its textbox structure.  When user presses soft backspace
>> (or any other soft key), they first apply the change to their model, and
>> then run a diffing algorithm against the previous model to generate IME
>> commands to send to the editor process for reconciliation.  For backspace,
>> this command is typically deleteSurroundingText(0, 1), but depending on the
>> vagaries of the diffing algorithm, it's sometimes much more complex.  (Note
>> that the AOSP keyboard source doesn't appear to illustrate this,
>> unfortunately.)
>>
>
> Right, but then the JavaScript editor takes over and mangles the DOM in
> some different way than what the IME expects and then nothing really works.
> As far as I understand,. you guys actually tell the keyboard producers that
> they need to be aware of DOm changes that they haven't introduced
> themselves and that they should subscribe to updates to the text.
>
> It seems IMEs are mainly a headache for JS devs because they cannot be
> canceled and until now it was also difficult to know what the user tried to
> do. If the editor needs to do its own DOM updates , for example because we
> are in a collaborative editor, the problem is getting worse if the IEM is
> just continous with no interruptable events and never reading back the
> actual DOM. So we should keep that to a minimum.
>
>
>>
>> This does make things a lot more complicated for JS editors as they will
>>> have to wait for the end of the composition and then potentially undo
>>> everything that was doen during the composition. So we are back at diffing
>>> the DOM.
>>>
>>
>>> There is one case that we cannot really seem to address as long as IME
>>> is the black box it is: If a user selects text from paragraph 1 to 3 and
>>> then hits a signle letter on the keyboard, the browser just merges those
>>> paragraphs. This is of coruse resolved in level 2 of the spec, but given
>>> that you don't want to implement that level, I guess the only other way is
>>> to giving more control over the IME to JS itself.
>>>
>>
>>>
>
>
> If the problem here was only a misunderstanding of the usage of the
> inputTypes, and Chromium can be changed to be spec compliant in the choice
> of input type (level 1 compliance), then there really isn't any need to
> keep the difference in cancelability for all these other input types. The
> difference between level 1 and level 2 of the spec would then simply boil
> down to level 2 having 4 IME-related inputTypes and level 1 having just 1.
>
>
>
>
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [1] https://github.com/klausw/hackerskeyboard/blob/3cefa2f89
>>> 797d5c55d49b900d8caed7074582ef0/java/src/org/pocketworkstati
>>> on/pckeyboard/LatinIME.java#L2137
>>>
>>>
>>> [2] https://developer.android.com/reference/android/view/inp
>>> utmethod/InputConnection.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jun 27, 2017 at 2:04 AM, Johannes Wilm <mail@johanneswilm.org>
>>> wrote:
>>>
>>>> Hey Alexandre,
>>>> as far as our agreement hitherto had been, only a few of these will
>>>> ever be triggered during IME, and those are exclusive to IME. So in other
>>>> words: those input types that are used in IME are not used outside of it
>>>> and vice versa. I understand that on Android, a lot of things have been
>>>> wrapped in IME, which seems like a different problem that we hopefully will
>>>> resolve once we get more control over the IME process.
>>>>
>>>> In effect it would mean that on Android, most of the inputTypes would
>>>> likely not be in use. For example -- on some desktop OSes there are special
>>>> keyboard shortcuts for operations such as deleting from the caret position
>>>> to the next softline wrapping. On other OSes without such a keyboard
>>>> shortcut, these inputTypes would not be triggered at all.
>>>>
>>>> > because "Paragraph break" and "Delete word backward" are plain-text
>>>> operations
>>>>
>>>> Not necessarily. If a word is partially styled, then deleting that word
>>>> may mean several other DOM changes. SImilarly, if the caret is in an
>>>> italic-styled part of a line that is not fully styled, deleting to the end
>>>> of the line will not just be a plaintext operation.
>>>>
>>>>
>>>> The examples I mentioned were just randomly chosen. We'll need to build
>>>> a strong argument fro JS devs to be using the beforeinput event, and the
>>>> part about why some things can be canceled while others cannot still seems
>>>> a bit weak.
>>>>
>>>> On Tue, Jun 27, 2017 at 12:40 AM, Alexandre Elias <aelias@chromium.org>
>>>> wrote:
>>>>
>>>>> The philosophy is indeed that an IME might send it, and we erred a bit
>>>>> on the side of conservatism on the list to avoid locking ourselves in a bad
>>>>> situation if an IME does wind up sending it.  IMEs do not understand rich
>>>>> text nor the clipboard so it was safe to exclude ordered lists and "cut".
>>>>> But they have a wide variety of APIs to alter plain text, and because
>>>>> "Paragraph break" and "Delete word backward" are plain-text operations (in
>>>>> contenteditable, simple plain-text carriage returns might be mapped to
>>>>> <p>), it's possible one of them might send it.
>>>>>
>>>>> I'm not totally opposed to making these 2 cancelable if there is a
>>>>> clear use case and we can guarantee that we never map IME operations to it
>>>>> but to an alternate event, though.  Or if the problem is mostly confusion
>>>>> rather than a use case, we can explain the reason is "plain textiness ~=
>>>>> IME".
>>>>>
>>>>>
>>>>> On Mon, Jun 26, 2017 at 2:17 PM, Rick Byers <rbyers@chromium.org>
>>>>> wrote:
>>>>>
>>>>>> +aelias.  Personally I really hope we can achieve interop around all
>>>>>> this :-)
>>>>>>
>>>>>> On Mon, Jun 26, 2017 at 4:43 PM, Johannes Wilm <mail@johanneswilm.org
>>>>>> > wrote:
>>>>>>
>>>>>>> Hey,
>>>>>>> the TAG review of the input events spec asked us among other things
>>>>>>> to convert the long list of inputType values to a table. I have now done
>>>>>>> that for both levels. Now that I am done I get a clearer picture of which
>>>>>>> iof the inputTypes the Google people decided to not make cancelable in the
>>>>>>> level 1 version. They are marked as "Undefined" so that also those
>>>>>>> implementing level 2 are compliant with level 1. [1]
>>>>>>>
>>>>>>> I wonder though -- what made you guys decide to make exactly those
>>>>>>> not be cancelable? For example, inserting an ordered list can be canceled,
>>>>>>> but inserting a paragraph break cannot be canceled. Neither one of these
>>>>>>> should be happening during an IME composition, right? Or if I hit a special
>>>>>>> key combination to delete a word backward, that cannot be canceled. But if
>>>>>>> instead I select the word and hit the key combination for cut, that is
>>>>>>> cancelable.
>>>>>>>
>>>>>>> Is this list meant to be final or is this still work in progress? If
>>>>>>> this is final, then maybe it would be a good idea to write up soem
>>>>>>> explanation fro the philosophy that lies behind these choices. Otherwise I
>>>>>>> am sure a lot of JS devs will be just as confused as me about this.
>>>>>>>
>>>>>>>
>>>>>>> [1] https://rawgit.com/w3c/input-events/v1/index.html#h-inte
>>>>>>> rface-inputevent-attributes
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Johannes Wilm
>>>>>>> http://www.johanneswilm.org
>>>>>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Johannes Wilm
>>>> http://www.johanneswilm.org
>>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>>
>>>
>>>
>>>
>>> --
>>> Johannes Wilm
>>> http://www.johanneswilm.org
>>> tel: +1 (520) 399 8880 <(520)%20399-8880>
>>>
>>
>>
>
>
> --
> Johannes Wilm
> http://www.johanneswilm.org
> tel: +1 (520) 399 8880 <(520)%20399-8880>
>



-- 
Johannes Wilm
http://www.johanneswilm.org
tel: +1 (520) 399 8880