Re: 2 Proposals for Minimum Viable InputEvent

Sorry for jumping in a bit out of context, but if beforeinput will not be cancelable (initially?) then why not just add the missing context (e.g., what text _was_ replaced/inserted and where) into the 'input' event? (And have 'input' dispatch a derived event type with that info.)


________________________________
From: Ojan Vafai <ojan@google.com>
Sent: Tuesday, February 7, 2017 8:18:49 PM
To: Johannes Wilm; Alexandre Elias
Cc: Dave Tapuska; Chong Zhang; public-editing-tf@w3.org
Subject: Re: 2 Proposals for Minimum Viable InputEvent

Oh, and #5, our intention is not to violate the spec, but rather to get the spec changed so that all browsers can agree on behavior here. Even in the case that Alexandre mentioned where iOS will deliver two events and Android one, the event types would be the same (namely ‘insertCompositionText’). So the interoperability impact will be fairly minimal.

The big difference here is that for IMEs that don't change the markup at start of composition, they're not forced to first delete the text and reinsert it because of having a delete followed by an insert for what is essentially just an insert.

On Tue, Feb 7, 2017 at 8:12 PM Ojan Vafai <ojan@google.com<mailto:ojan@google.com>> wrote:
This discussion has gotten rather complicated, but I want to call out a few points.

  1.  In all these proposals, JS always knows about changes to the DOM before they happen via the beforeInput event and some of those cases are cancellable. In the latest proposal that started this thread, we're proposing that more of those cases are not cancellable. We think this will lead to a simpler system for web authors to understand (all insert*/delete* are not cancellable) and something that we can implement more efficiently.
  2.  One of the use cases for beforeInput is JS libraries that want to do the DOM modifications themselves. That's certainly not the only use case though. Right now on Android, there's no way to know which text is going to be replaced by which text when the user does an editing operation. All proposals here address that problem.
  3.  If we can find a way to make the insert*/delete* types cancellable, it will most likely be backwards-compatible to make them cancellable in the future. In the meantime, we can get a v1 of beforeInput out there now and solve the basic use case in #2.
  4.  The only way to make interaction with IMEs truly sane is to expose a first-class IME API to talk to IMEs. We've heard from JS library authors in this group that they aren't interested in adding that complexity to their libraries anytime soon. So, we're not prioritizing designing those API. The sad reality is that beforeInput will have to have some warts due to IME interaction on *desktop* platforms not allowing cancellation. We believe the new proposal reduces warts by simplifying what is or isn't cancellable in a way where a naive developer is more likely to support non-latin languages by accident. This is obviously a subjective intuition, so I doubt we'll get anywhere by arguing about whether this is a valid belief.

On Mon, Feb 6, 2017 at 7:45 PM Johannes Wilm <johannes@fiduswriter.org<mailto:johannes@fiduswriter.org>> wrote:


On Tue, Feb 7, 2017 at 4:00 AM, Alexandre Elias <aelias@google.com<mailto:aelias@google.com>> wrote:
On Mon, Feb 6, 2017 at 6:18 PM, Johannes Wilm <johannes@fiduswriter.org<mailto:johannes@fiduswriter.org>> wrote:
On Tue, Feb 7, 2017 at 12:23 AM, Alexandre Elias <aelias@google.com<mailto:aelias@google.com>> wrote:
Hi, I'm the maintainer of IME on Android.  I'll be happy to attend the conference call next week.  As you've surmised, Android does need to perform a DOM change to merge two nodes "later".  However, I disagree that those 5 steps are somehow needed.  Our plan is to send a single insertCompositionText beforeinput event instead of 5 events.

Nice to meet you Alexandre!

It's a bit unfrotunate that you haven't been able to participate in the last 3 years of discussions about this subject (I think some people have even been discussing this for more than 8 years), and it seems that the mailing list isn't as alive any more as it once used to be, but I'll do my best to give the reasoning why we came to the conclusion that these events are needed. I would also recommend consulting the various open and closed issues on github and to consult the archive of the mailing list. Also, various Google people have been behind a lot if not most of the various various concepts, so they can probably give you more background about this as well.


Arguments why each is not needed:

 > 1. deleteByComposition --This step is only needed when recomposing. A composed word can go across element boundaries, so this may delete more than just contents of a text node. (fx " He<b>ll</b><i>o fish</i>!"). This step needs to be executed at the moment your Android IME changes the DOM, whether this is at the moment the user taps the word and the underline appears or at a later stage.

There's no need for a separate "delete" event, because insertion-to-replace-range is general enough to represent a deletion range as well.  I prefer to reserve "delete" events exclusively for intended IME deletions (e.g. select word and press backspace).

But I am a little confused: From the proposal Chong sent in, it sounded like you just wouldn't have any beforeinput event for this initial deletion. That would create a range of different problems, such as:

I agree we should not hide any editing information from JS.  I am saying that event 2/3 can encompass the deletion.  An implied, wrong assumption underlying your argument seems to be that an insertion operation can only ever add new text but not remove existing text.


No, that must be a misunderstanding. I don't know why you would think that. As I said previsouly, if  beforeinput events during IME are cancelable, 1 (deletion) and 2 (addition cshould be combinable into one single event,


I don't quite get what you mean by combining steps two and three. Step two was the reinsertion of the initial composition string into the DOM. Step three is the change during the composition. There will usually be many step threes, but only one step two.

The reason there is a step one and two at all is that the IME converts the composition string from being something that can contain a complex html-structure to just be  a plaintext string (in most cases anyway).

So step 1 will remove "He<b>l</b>p" and step 2 will re-add it as "Help". Combining step 1 + 2 should work if step 2 is also cancelable. Combining step 2 and 3, or maybe 1 and 2 and 3 (?) would mean that we go straight from "He<b>l</b>p" to "Hel". and then the next two step 3 would add "l" and "o". In many cases the JS will probably be able to guess what the individual parts of this combination of 1/2/3 initially were (what part was just about turning it into a plaintext and which part was then about modifying the plaintext), but it would seem a bit meshed and I'm not sure one always can determine it. Nevertheless, if iME input is cancelable, then this does sound better than what was on the table previosuly.



  In reality, insertions are always replacements of the content of the caret/selection/composition with text (for example, if you select text with the mouse and then type "a" on a physical keyboard, no deletion event is sent).

right. That'sw hy we have targetranges. But the difference for keyboard inout is that it is cancelable and there is no automatic normalization to plaintext. This is different than the situation with IME.


* The more advanced editors today are moving toward a model where they hold a json version of the document which is separate from the DOM. They only make adjustments to the DOM when needed, and if the user changes something about the DOM, they make adjustments to the  json document. The json version is the authoritative copy of the document. For this to work, they need to know about any change to the DOM. If you make user-initiated DOM changes that are not causing any beforeinput event at all, the beforeinput event is somewhat useless, aand they will have to continue to monitor user initiated DOM-changes in other ways.

* In a collaborative editor, if you make a change to the DOM of one user, and the JS is not being made aware of this change, then it won't notify the collaborators of this change, and so the collaborator will ahve different versions of the document. Again, the point of beforeinput events will be gone. JS develoeprs will use mutation observers or some other technology instead.




> 2. insertCompositionText -- Also this step only needs to happen at the moment only for recompositions. It reinserts the word from step one, but this time as plaintext in one element (fx "Hello" => "<i>Hello fish</i>")

On Android because of the "late" timing, this merge operation would happen in the same message loop task as step 3, so there is no need for this event.  We prefer to go directly from "He<b>ll</b><i>o fish</i>" -> "<i>Help<i>".


Yes, in a perfect world, where all IME-related events are cancelable, this could be done and would be preferable, as for example Yosin has pointed out ( https://github.com/w3c/input-events/issues/45#issuecomment-261427648 ). We had a long discussion during the fall of 2015 about this, and among other ones several developers from Chromium eventually convinced us here that IME-events cannot be guaranteed to be cancelable. Even moving the caret to somewhere else during the composition may somehow crash the IME (hwo this is possible is a bit of a mystery to me as a JS developer).

One could then say ok, combine 1 and 2 and make them both non-cancelable. The problem with that is that the browser doesn't always know the seamntic meaning of the various DOM elements used within a document and it may end up making decisions of what to delete that are problematic.

I've been trying to figure out where these claims about IME from some Chromium team members came from exactly and haven't figured it out, nobody has a precise memory.  They seem to be based on a vague fears rather than concrete knowledge of how IMEs behave.  I agree with your instinct that the assumptions behind it don't make too much sense, that's why I think we should hold off before we commit to a complicated model based on those assumptions.

Well, maybe you guys need to discuss that internally. Until now what we said here is that we will assume the worst case scenario (that IMEs not being allowed to directly modify the DOM will lead to them crashing) and then in a future version to hopefully get one that doesn't crash.

Btw -- just to make it clear because this seems to have been a common misconception between JS developers and browser developers in the past: When we ask for these things to be cancelable, in 80% of cases this is not because we want to stop the user from doing things. It's more than we want to do the corresponding DOM changes in JS code, because they are specific to the JS editor the user is currently using and they may not necessarily make sense to the browser.






However, rniwa@apple.com<mailto:rniwa@apple.com> has told us that on OS X, the standard platform behavior is to merge the nodes at time of composition start.  Therefore, on OS X step 2 is still needed and there would be 2 beforeinput events in total, but on Android there would be only 1.  This represents a substantive platform difference that I think it makes sense to allow to differ in the spec.  I don't think we should try to artificially synchronize the platforms by always sending 2 events -- on Android this would simply result in unnecessary work and side effects that would never be seen by the user.

I think you should instead consider making this two events on Android. Android is really great, but text editing in Chrome on Android is really problematic especially when it comes to IME recomposition, as I am sure you are aware. I have tried various keyboard, and but even on mainstream sites such as Facebook, using a Google-provided keyboard, recomposition tends to just mess up entire sentences, throw random words around, deleting or merging words that are close to what one is trying to recompose, etc.

Right, we are in process of fixing all of these bugs and expect to get the remaining ones fixed in M58/M59 timeframe.  Anyway, this doesn't address my point.  Personally, I believe that firing additional unnecessary events will prove harmful to both UAs and JS editors -- in reality an atomic IME change that just happened, but you are proposing to drip-feed it into many separate events.  Your proposal doesn't provide JS any additional information (it just splits information into pieces), so therefore I don't think it can possibly be needed for any use case either.


Speed is not as much a concern anyway because it all depends on the human who is interacting with it, but in addition these really major issues mean that recomposition as it exists on Android as of today is not very useful. In addition there are the various issues that JS developers have complained about for a while: no compositionstart event, etc. . I am sure you know the list of grievances with it all too well.

We should always be firing compositionStart today.  Feel free to file a bug against me if you know of a case where we don't.

I recently sent a list of prioritized bugs from several JS editor projects to the Chromium team that they had requested from em. I assume they forwarded those issues relevant to Android to you.


Not knowing the details, but I could imagine you can check for event listeners, and only if those are present you make it two events.

But as an alternative, and to try to get a bit closer to both your and Yosin's position, how about this:  We make two IME models, one for IMEs where the beforeinput events can be cancelled in which step 1 and 2 are merged and one where they cannot be cancelled where they are separate.

Disagree, we should keep things as simple as possible.  I want us to fire a minimum of events and avoid complicated logic forking.  I don't want us to introduce permanent complexity into the platform for the sake of a short-term compromise.

Well, you said that Apple does one thing and you do another. So either the two of you can agree or you don't. If you cannot agree, we'll have to have two different models.

In theis scenario, if the Chrome model consists of just cutting out 4 of 5 events on Android, then as I tried to explain in the last few emails and the comments on the proposal doc, the event will no longer function to build an editor on top of. At least not without combining it with mutation observers and other technologies.

Having a model where IME events are cancelable is really something we all hoped to do in a future version of the spec. If the situation is such that Chromium/Android would prefer to do this right now, whereas Apple prefers to stay with the non-cancelable IME-model then that would not seem like terribly different models.




> 4. deleteCompositionText -- removed (fx "help"): It removed the text from the underline and it may not be the entire composition.

> 5. insertFromComposition -- adds "help" permanently, and gives the JS the opposrtunity to rebuild the complex HTML structure lost in step 1.

Step 3 is already permanent.  The fact that there is still an active composition underline at the time of step 3 does not make it somehow temporary.  It has already been injected into the DOM and caused all side effects.  So the JS should rebuild its complex structure at time of event 3, and events 4/5 are redundant.

When a composition starts, they temporarily pause most activities because they are not really allowed to intervene. Then, once the composition is finished, they deal with the finished text.  input. For example, in step 5 they may reapply some partial styling that was lost in step 1.

"compositionend" event is fired at the same timing that steps 4/5 would be, isn't it sufficient for this use case?

There is one problem in that compositionend with data="" could mean the composition has been cancelled or that the final compositionstring is empty. The distinction between 4 and 5 is needed if 2, 3 & 4 are non-cancelable. If they are cancelable, then one should be able to combine 1 and 2, and skip 4 and 5 and the JS would continously make any adjustments it needs to the DOM (while taking into consideration what the compositionstring is at any moment).







The reason you state for the proposed change is that "On Android it’s very common for IME to start composition whenever a cursor is moved onto a word.  This is primarily intended to passively show an underline to the user (to indicate what would be replaced *if* user selects an alternate spelling)."

If such tapping leads to actual DOM changes, then  the JS needs to know about those(step 1/2). But if you don't change the DOM immediately and only do so at a later stage when you "really" start the composition, then you can also wait until that stage.


So these steps are needed on allplattforms, no matter whether or not you change how IMEs work on Android.


Adding aelias@ to make sure he is able to make the call on the 14th.

dave.

On Fri, Feb 3, 2017 at 1:33 AM, Johannes Wilm <johannes@fiduswriter.org<mailto:johannes@fiduswriter.org>> wrote:
Could we get the person or people who came up with these changes due to Android to participate in the call on the 14th? I think it's very important to have them invovled in this discussion.

On Fri, Feb 3, 2017 at 10:21 AM, Johannes Wilm <johannes@fiduswriter.org<mailto:johannes@fiduswriter.org>> wrote:

On Fri, Feb 3, 2017 at 10:18 AM, Johannes Wilm <johannes@fiduswriter.org<mailto:johannes@fiduswriter.org>> wrote:
...
Right now I cannot see any purpose of the beforeinput event with these changes applied, but maybe I am missing something?


I need to restarct that. I guess it could still be useful to stop native bold/italic buttons from making their own, non-controlled DOM changes. It's just for text input that it is irrelevant. And IME will continue to be a mess.

--
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org<http://www.fiduswriter.org/>



--
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org<http://www.fiduswriter.org/>




--
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org<http://www.fiduswriter.org/>




--
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org<http://www.fiduswriter.org/>




--
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org<http://www.fiduswriter.org/>