Re: 2 Proposals for Minimum Viable InputEvent

On Wed, Feb 8, 2017 at 6:41 AM, Ojan Vafai <ojan@google.com> wrote:

> My preference is that we move forward with making the subset of insert and
> delete types I suggested in my earlier email uncancellable in v1 and then
> do a proper exploration of what platforms and contexts we could potentially
> make all inputTypes cancellable.
>

> Up until now, we've mostly worked off lore of what's possible. With one
> exception, Masayuki from Mozilla explained to us that the latest iteration
> of IME APIs on Windows are never cancellable IIRC.
>
> It may be the case that once we have a good grasp of the situation on all
> platforms we would reconsider whether we could make all eventTypes
> cancellable on some platforms.
>

I can see how you guys don't want to send "phantom events" on IME on
Android. Fair enough, and the events you do send may be non-cancelable
until you figure out if you can make them cancelable on Android. This
mainly seems to be  about making a clarification that preventdefaulting the
beforeinput event during IME is not ever meant to stop the IME and that the
JS is aware that the IME is continuing and that it has to behave
accordingly. This is something which we have discussed here quite a bit
already and something that you showed working with the long-press IME on
Chrome/Mac OS X last summer.

So that part I think we can work with.

What is much more problematic is that you want to remove cancelability for
almost all beforeinput events.

Under certain circumstances there is currently an incentive to let changes
to text nodes go through directly [1]. One of the editors, ProseMirror
therefore recently changed to a model where it would let the browser handle
changes to text nodes and then roll-back those changes if undesired. The
problem was that this lead to "edit flashes" [2] where the user can see the
DOM changes twice rapidly. If someone was to build an editor based on this
non-cancelable beforeinput event, one would continue to diff the DOM and be
doomed to rollback all undesired changes rather than being able to prevent
them from happening in the first place and see this edit flashes.

This will have a crippling effect on the event and will have the practical
consequence is that the event will not be used in most cases as developers
will use existing events and methods, where they only roll-back if they
really have to. Beforeinput will be degraded to be about finding out where
the caret was at the beginning of the operation if one doesn't know this
already.

If the idea with this "crippling" is to try to get JS developers to write
code that automatically works for IME as well, that is just  not going to
happen: developers could already now likely run most of the mobile version
of their app on the desktop, but they don't because it is so limited. So
they use the best options they have available on each platform.







[1] If one wants the browser-internal spell checker to underline more or
less correctly (looking at a way of marking which ranges have been
spellchecked and which ones still need spell checking is something I hhoped
we could look at once beforeinput is out the door).

[2]
https://discuss.prosemirror.net/t/what-does-a-read-only-editor-even-mean/221/29


>
> On Tue, Feb 7, 2017 at 8:48 PM Yoshifumi Inoue <yosin@google.com> wrote:
>
>>
>> >It's a bit unfrotunate that you haven't been able to participate in the
>> last 3 years of discussions about this subject (I think some people have
>> >even been discussing this for more than 8 years), and it seems that the
>> mailing list isn't as alive any more as it once used to be, but I'll do my
>> >best to give the reasoning why we came to the conclusion that these events
>> are needed. I would also recommend consulting the various open >and closed
>> issues on github and to consult the archive of the mailing list. Also,
>> various Google people have been behind a lot if not most of the >various
>> various concepts, so they can probably give you more background about this
>> as well.
>>
>> Alexandre has been backed up, kochi@, ex-W3C IME editor, koji@, and
>> yosin@. And, we conclude "deleteCompositionText" is redundant.
>> Personally, I'm still seeking perfect IME world. I think non-canablable
>> IME behavior is big technical debt and should resolve for next generation.
>>
>>
>>
>>
>>
>> 2017年2月8日(水) 13:28 Ojan Vafai <ojan@google.com>:
>>
>> It let's you know before the DOM has been modified so you can
>> appropriately track changes. The targetRange can't serve that purpose after
>> the DOM has been modified since it's a position in the DOM that no longer
>> exists, not just a (rich)text string.
>>
>> I do think we should add the new fields (e.g. inputType) to the input
>> event as well to make beforeInput and input as similar as possible.
>>
>>
>> On Tue, Feb 7, 2017 at 8:22 PM Travis Leithead <
>> travis.leithead@microsoft.com> wrote:
>>
>> Sorry for jumping in a bit out of context, but if beforeinput will not be
>> cancelable (initially?) then why not just add the missing context (e.g.,
>> what text _was_ replaced/inserted and where) into the 'input' event? (And
>> have 'input' dispatch a derived event type with that info.)
>>
>>
>> ------------------------------
>> *From:* Ojan Vafai <ojan@google.com>
>> *Sent:* Tuesday, February 7, 2017 8:18:49 PM
>> *To:* Johannes Wilm; Alexandre Elias
>> *Cc:* Dave Tapuska; Chong Zhang; public-editing-tf@w3.org
>> *Subject:* Re: 2 Proposals for Minimum Viable InputEvent
>>
>> Oh, and #5, our intention is not to violate the spec, but rather to get
>> the spec changed so that all browsers can agree on behavior here. Even in
>> the case that Alexandre mentioned where iOS will deliver two events and
>> Android one, the event types would be the same (namely
>> ‘insertCompositionText’). So the interoperability impact will be fairly
>> minimal.
>>
>> The big difference here is that for IMEs that don't change the markup at
>> start of composition, they're not forced to first delete the text and
>> reinsert it because of having a delete followed by an insert for what is
>> essentially just an insert.
>>
>> On Tue, Feb 7, 2017 at 8:12 PM Ojan Vafai <ojan@google.com> wrote:
>>
>> This discussion has gotten rather complicated, but I want to call out a
>> few points.
>>
>>    1. In all these proposals, JS always knows about changes to the DOM
>>    before they happen via the beforeInput event and some of those cases are
>>    cancellable. In the latest proposal that started this thread, we're
>>    proposing that more of those cases are not cancellable. We think this will
>>    lead to a simpler system for web authors to understand (all insert*/delete*
>>    are not cancellable) and something that we can implement more efficiently.
>>    2. One of the use cases for beforeInput is JS libraries that want to
>>    do the DOM modifications themselves. That's certainly not the only use case
>>    though. Right now on Android, there's no way to know which text is going to
>>    be replaced by which text when the user does an editing operation. All
>>    proposals here address that problem.
>>    3. If we can find a way to make the insert*/delete* types
>>    cancellable, it will most likely be backwards-compatible to make them
>>    cancellable in the future. In the meantime, we can get a v1 of beforeInput
>>    out there now and solve the basic use case in #2.
>>    4. The only way to make interaction with IMEs truly sane is to expose
>>    a first-class IME API to talk to IMEs. We've heard from JS library authors
>>    in this group that they aren't interested in adding that complexity to
>>    their libraries anytime soon. So, we're not prioritizing designing those
>>    API. The sad reality is that beforeInput will have to have some warts due
>>    to IME interaction on *desktop* platforms not allowing cancellation. We
>>    believe the new proposal reduces warts by simplifying what is or isn't
>>    cancellable in a way where a naive developer is more likely to support
>>    non-latin languages by accident. This is obviously a subjective intuition,
>>    so I doubt we'll get anywhere by arguing about whether this is a valid
>>    belief.
>>
>>
>> On Mon, Feb 6, 2017 at 7:45 PM Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>>
>>
>>
>> On Tue, Feb 7, 2017 at 4:00 AM, Alexandre Elias <aelias@google.com>
>> wrote:
>>
>> On Mon, Feb 6, 2017 at 6:18 PM, Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>>
>> On Tue, Feb 7, 2017 at 12:23 AM, Alexandre Elias <aelias@google.com>
>> wrote:
>>
>> Hi, I'm the maintainer of IME on Android.  I'll be happy to attend the
>> conference call next week.  As you've surmised, Android does need to
>> perform a DOM change to merge two nodes "later".  However, I disagree that
>> those 5 steps are somehow needed.  Our plan is to send a single
>> insertCompositionText beforeinput event instead of 5 events.
>>
>>
>> Nice to meet you Alexandre!
>>
>> It's a bit unfrotunate that you haven't been able to participate in the
>> last 3 years of discussions about this subject (I think some people have
>> even been discussing this for more than 8 years), and it seems that the
>> mailing list isn't as alive any more as it once used to be, but I'll do my
>> best to give the reasoning why we came to the conclusion that these events
>> are needed. I would also recommend consulting the various open and closed
>> issues on github and to consult the archive of the mailing list. Also,
>> various Google people have been behind a lot if not most of the various
>> various concepts, so they can probably give you more background about this
>> as well.
>>
>>
>>
>> Arguments why each is not needed:
>>
>>  > 1. deleteByComposition --This step is only needed when recomposing. A
>> composed word can go across element boundaries, so this may delete more
>> than just contents of a text node. (fx " He<b>ll</b><i>o fish</i>!"). This
>> step needs to be executed at the moment your Android IME changes the DOM,
>> whether this is at the moment the user taps the word and the underline
>> appears or at a later stage.
>>
>> There's no need for a separate "delete" event, because
>> insertion-to-replace-range is general enough to represent a deletion range
>> as well.  I prefer to reserve "delete" events exclusively for intended IME
>> deletions (e.g. select word and press backspace).
>>
>>
>> But I am a little confused: From the proposal Chong sent in, it sounded
>> like you just wouldn't have any beforeinput event for this initial
>> deletion. That would create a range of different problems, such as:
>>
>>
>> I agree we should not hide any editing information from JS.  I am saying
>> that event 2/3 can encompass the deletion.  An implied, wrong assumption
>> underlying your argument seems to be that an insertion operation can only
>> ever add new text but not remove existing text.
>>
>>
>>
>> No, that must be a misunderstanding. I don't know why you would think
>> that. As I said previsouly, if  beforeinput events during IME are
>> cancelable, 1 (deletion) and 2 (addition cshould be combinable into one
>> single event,
>>
>>
>> I don't quite get what you mean by combining steps two and three. Step
>> two was the reinsertion of the initial composition string into the DOM.
>> Step three is the change during the composition. There will usually be many
>> step threes, but only one step two.
>>
>> The reason there is a step one and two at all is that the IME converts
>> the composition string from being something that can contain a complex
>> html-structure to just be  a plaintext string (in most cases anyway).
>>
>> So step 1 will remove "He<b>l</b>p" and step 2 will re-add it as "Help".
>> Combining step 1 + 2 should work if step 2 is also cancelable. Combining
>> step 2 and 3, or maybe 1 and 2 and 3 (?) would mean that we go straight
>> from "He<b>l</b>p" to "Hel". and then the next two step 3 would add "l" and
>> "o". In many cases the JS will probably be able to guess what the
>> individual parts of this combination of 1/2/3 initially were (what part was
>> just about turning it into a plaintext and which part was then about
>> modifying the plaintext), but it would seem a bit meshed and I'm not sure
>> one always can determine it. Nevertheless, if iME input is cancelable, then
>> this does sound better than what was on the table previosuly.
>>
>>
>>
>>
>>   In reality, insertions are always *replacements* of the content of the
>> caret/selection/composition with text (for example, if you select text with
>> the mouse and then type "a" on a physical keyboard, no deletion event is
>> sent).
>>
>>
>> right. That'sw hy we have targetranges. But the difference for keyboard
>> inout is that it is cancelable and there is no automatic normalization to
>> plaintext. This is different than the situation with IME.
>>
>>
>>
>> * The more advanced editors today are moving toward a model where they
>> hold a json version of the document which is separate from the DOM. They
>> only make adjustments to the DOM when needed, and if the user changes
>> something about the DOM, they make adjustments to the  json document. The
>> json version is the authoritative copy of the document. For this to work,
>> they need to know about any change to the DOM. If you make user-initiated
>> DOM changes that are not causing any beforeinput event at all, the
>> beforeinput event is somewhat useless, aand they will have to continue to
>> monitor user initiated DOM-changes in other ways.
>>
>> * In a collaborative editor, if you make a change to the DOM of one user,
>> and the JS is not being made aware of this change, then it won't notify the
>> collaborators of this change, and so the collaborator will ahve different
>> versions of the document. Again, the point of beforeinput events will be
>> gone. JS develoeprs will use mutation observers or some other technology
>> instead.
>>
>>
>>
>>
>>
>> > 2. insertCompositionText -- Also this step only needs to happen at the
>> moment only for recompositions. It reinserts the word from step one, but
>> this time as plaintext in one element (fx "Hello" => "<i>Hello fish</i>")
>>
>> On Android because of the "late" timing, this merge operation would
>> happen in the same message loop task as step 3, so there is no need for
>> this event.  We prefer to go directly from "He<b>ll</b><i>o fish</i>" ->
>> "<i>Help<i>".
>>
>>
>>
>> Yes, in a perfect world, where all IME-related events are cancelable,
>> this could be done and would be preferable, as for example Yosin has
>> pointed out ( https://github.com/w3c/input-events/issues/45#issuecomment-
>> 261427648 ). We had a long discussion during the fall of 2015 about
>> this, and among other ones several developers from Chromium eventually
>> convinced us here that IME-events cannot be guaranteed to be cancelable.
>> Even moving the caret to somewhere else during the composition may somehow
>> crash the IME (hwo this is possible is a bit of a mystery to me as a JS
>> developer).
>>
>> One could then say ok, combine 1 and 2 and make them both non-cancelable.
>> The problem with that is that the browser doesn't always know the seamntic
>> meaning of the various DOM elements used within a document and it may end
>> up making decisions of what to delete that are problematic.
>>
>>
>> I've been trying to figure out where these claims about IME from some
>> Chromium team members came from exactly and haven't figured it out, nobody
>> has a precise memory.  They seem to be based on a vague fears rather than
>> concrete knowledge of how IMEs behave.  I agree with your instinct that the
>> assumptions behind it don't make too much sense, that's why I think we
>> should hold off before we commit to a complicated model based on those
>> assumptions.
>>
>>
>> Well, maybe you guys need to discuss that internally. Until now what we
>> said here is that we will assume the worst case scenario (that IMEs not
>> being allowed to directly modify the DOM will lead to them crashing) and
>> then in a future version to hopefully get one that doesn't crash.
>>
>> Btw -- just to make it clear because this seems to have been a common
>> misconception between JS developers and browser developers in the past:
>> When we ask for these things to be cancelable, in 80% of cases this is not
>> because we want to stop the user from doing things. It's more than we want
>> to do the corresponding DOM changes in JS code, because they are specific
>> to the JS editor the user is currently using and they may not necessarily
>> make sense to the browser.
>>
>>
>>
>>
>>
>>
>>
>>
>> However, rniwa@apple.com has told us that on OS X, the standard platform
>> behavior is to merge the nodes at time of composition start.  Therefore, on
>> OS X step 2 is still needed and there would be 2 beforeinput events in
>> total, but on Android there would be only 1.  This represents a substantive
>> platform difference that I think it makes sense to allow to differ in the
>> spec.  I don't think we should try to artificially synchronize the
>> platforms by always sending 2 events -- on Android this would simply result
>> in unnecessary work and side effects that would never be seen by the user.
>>
>>
>> I think you should instead consider making this two events on Android.
>> Android is really great, but text editing in Chrome on Android is really
>> problematic especially when it comes to IME recomposition, as I am sure you
>> are aware. I have tried various keyboard, and but even on mainstream sites
>> such as Facebook, using a Google-provided keyboard, recomposition tends to
>> just mess up entire sentences, throw random words around, deleting or
>> merging words that are close to what one is trying to recompose, etc.
>>
>>
>> Right, we are in process of fixing all of these bugs and expect to get
>> the remaining ones fixed in M58/M59 timeframe.  Anyway, this doesn't
>> address my point.  Personally, I believe that firing additional unnecessary
>> events will prove harmful to both UAs and JS editors -- in reality an
>> atomic IME change that just happened, but you are proposing to drip-feed it
>> into many separate events.  Your proposal doesn't provide JS any additional
>> information (it just splits information into pieces), so therefore I don't
>> think it can possibly be needed for any use case either.
>>
>>
>>
>>
>> Speed is not as much a concern anyway because it all depends on the human
>> who is interacting with it, but in addition these really major issues mean
>> that recomposition as it exists on Android as of today is not very useful.
>> In addition there are the various issues that JS developers have complained
>> about for a while: no compositionstart event, etc. . I am sure you know the
>> list of grievances with it all too well.
>>
>>
>> We should always be firing compositionStart today.  Feel free to file a
>> bug against me if you know of a case where we don't.
>>
>>
>> I recently sent a list of prioritized bugs from several JS editor
>> projects to the Chromium team that they had requested from em. I assume
>> they forwarded those issues relevant to Android to you.
>>
>>
>>
>>
>> Not knowing the details, but I could imagine you can check for event
>> listeners, and only if those are present you make it two events.
>>
>> But as an alternative, and to try to get a bit closer to both your and
>> Yosin's position, how about this:  We make two IME models, one for IMEs
>> where the beforeinput events can be cancelled in which step 1 and 2 are
>> merged and one where they cannot be cancelled where they are separate.
>>
>>
>> Disagree, we should keep things as simple as possible.  I want us to fire
>> a minimum of events and avoid complicated logic forking.  I don't want us
>> to introduce permanent complexity into the platform for the sake of a
>> short-term compromise.
>>
>>
>> Well, you said that Apple does one thing and you do another. So either
>> the two of you can agree or you don't. If you cannot agree, we'll have to
>> have two different models.
>>
>> In theis scenario, if the Chrome model consists of just cutting out 4 of
>> 5 events on Android, then as I tried to explain in the last few emails and
>> the comments on the proposal doc, the event will no longer function to
>> build an editor on top of. At least not without combining it with mutation
>> observers and other technologies.
>>
>> Having a model where IME events are cancelable is really something we all
>> hoped to do in a future version of the spec. If the situation is such that
>> Chromium/Android would prefer to do this right now, whereas Apple prefers
>> to stay with the non-cancelable IME-model then that would not seem like
>> terribly different models.
>>
>>
>>
>>
>>
>>
>> > 4. deleteCompositionText -- removed (fx "help"): It removed the text
>> from the underline and it may not be the entire composition.
>>
>> > 5. insertFromComposition -- adds "help" permanently, and gives the JS
>> the opposrtunity to rebuild the complex HTML structure lost in step 1.
>>
>> Step 3 is already permanent.  The fact that there is still an active
>> composition underline at the time of step 3 does not make it somehow
>> temporary.  It has already been injected into the DOM and caused all side
>> effects.  So the JS should rebuild its complex structure at time of event
>> 3, and events 4/5 are redundant.
>>
>>
>> When a composition starts, they temporarily pause most activities because
>> they are not really allowed to intervene. Then, once the composition is
>> finished, they deal with the finished text.  input. For example, in step 5
>> they may reapply some partial styling that was lost in step 1.
>>
>>
>> "compositionend" event is fired at the same timing that steps 4/5 would
>> be, isn't it sufficient for this use case?
>>
>>
>> There is one problem in that compositionend with data="" could mean the
>> composition has been cancelled or that the final compositionstring is
>> empty. The distinction between 4 and 5 is needed if 2, 3 & 4 are
>> non-cancelable. If they are cancelable, then one should be able to combine
>> 1 and 2, and skip 4 and 5 and the JS would continously make any adjustments
>> it needs to the DOM (while taking into consideration what the
>> compositionstring is at any moment).
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The reason you state for the proposed change is that "On Android it’s
>> very common for IME to start composition whenever a cursor is moved onto a
>> word.  This is primarily intended to passively show an underline to the
>> user (to indicate what would be replaced *if* user selects an alternate
>> spelling)."
>>
>> If such tapping leads to actual DOM changes, then  the JS needs to know
>> about those(step 1/2). But if you don't change the DOM immediately and only
>> do so at a later stage when you "really" start the composition, then you
>> can also wait until that stage.
>>
>>
>> So these steps are needed on allplattforms, no matter whether or not you
>> change how IMEs work on Android.
>>
>>
>>
>> Adding aelias@ to make sure he is able to make the call on the 14th.
>>
>> dave.
>>
>> On Fri, Feb 3, 2017 at 1:33 AM, Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>>
>> Could we get the person or people who came up with these changes due to
>> Android to participate in the call on the 14th? I think it's very important
>> to have them invovled in this discussion.
>>
>> On Fri, Feb 3, 2017 at 10:21 AM, Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>>
>>
>> On Fri, Feb 3, 2017 at 10:18 AM, Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>> ...
>>
>> Right now I cannot see any purpose of the beforeinput event with these
>> changes applied, but maybe I am missing something?
>>
>>
>> I need to restarct that. I guess it could still be useful to stop native
>> bold/italic buttons from making their own, non-controlled DOM changes. It's
>> just for text input that it is irrelevant. And IME will continue to be a
>> mess.
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>>
>>
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>>
>>
>>
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>>
>>
>>
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>>
>>
>>
>>
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org