Re: 2 Proposals for Minimum Viable InputEvent

OK, thanks for the clarifications.

For me the bottom line I want to achieve is to have 1-to-1 mapping between
IME commands intended to mutate text and beforeinput+input pairs.  I don't
like the idea of splitting any one of them into fragmentary pieces nor of
generating phantom ones not corresponding to any text mutation.  That
generates unnecessary side effects and is difficult to reason about.

Sometimes this would cause "natural" platform differences based on the
different behavior of the actual IMEs, and that's fine because that likely
means a different user-facing UX that JS would also need to conform to.  On
Android in the replacement scenario described, there is exactly one
mutation where the nodes are both collapsed and replaced by the new word,
so there should be exactly one input event.  On Mac OS X there is first a
visible collapse at the beginning of the composition, then (later on) a
change to the new word, so there should be exactly two input events.  I'm
okay with such "natural" complexity but not with artificial
spec-originating complexity.

The idea of supporting a "preventDefault" on a fragmentary or phantom event
is particularly alarming to me, because I think "preventDefault" should
always be implemented by UAs as a simple early-return not performing some
action.  But the implication of "preventDefault" on a phantom event is that
the UA needs to proactively change the text when preventDefault is called
but do nothing when it isn't, which is weird and bound to lead to a cascade
of complexity.  In my way of seeing things, "cancelability" is not a sane
basis for fragmenting events, and the fragmenting is even counterproductive
to permitting workable cancelability.  (That's why I assumed the reason for
the split might be a "cannot-replace" assumption.)

>  I don't quite get what you mean by combining steps two and three. Step
two was the reinsertion of the initial composition string into the DOM.
Step three is the change during the composition. There will usually be many
step threes, but only one step two.

I simply meant combining step 2 into the first of the step 3s.

> There is one problem in that compositionend with data="" could mean the
composition has been cancelled or that the final compositionstring is empty.

I propose to solve it always sending another step-3-style
insertCompositionText("") before the compositionEnd in the case that the
text is cleared, and no input event in the cancellation case.  Then there
would be no need to ever look at the data field of compositionEnd.

> Btw -- just to make it clear because this seems to have been a common
misconception between JS developers and browser developers in the past:
When we ask for these things to be cancelable, in 80% of cases this is not
because we want to stop the user from doing things. It's more than we want
to do the corresponding DOM changes in JS code, because they are specific
to the JS editor the user is currently using and they may not necessarily
make sense to the browser.

That's useful context.  But my take from this is that beforeinput
preventDefaulting in *any* form cannot be used to meet this use case, at
least on Android.  Reason: on Android, we need to call the IME with
"restartInput()" whenever the text changes unexpectedly, which fully
reinitializes the IME and cancels ongoing compositions.  This call is not
at all desirable or needed if the JS editor is doing essentially the same
thing as the default implementation would have.  But unfortunately, there
is no sane way for UAs to know whether the JS is "trying" to stop the
action or if it's "trying" to recreate it.  We'd have to bias towards
oversending or undersending restartInput, likely via unreliable
heuristics.  The bugginess would be severe with either kind of bias.

An alternate proposal: what if, when JS modifies the DOM within the scope
of any "input" event, then the UA would run a before/after comparison to
verify whether the summation of JS changes wound up changing the plain text
distillation of the DOM (and therefore whether it's provable that the IME
can sit blissfully unaware that anything strange happened).  If the plain
text representation changed, then the UA would always send restartInput and
cancel composition, as UAs largely do in response to unexpected JS changes
today.  But if the only thing that changed was styling or other rich DOM
substructure, the UA would *not* send restartInput and it would also
reconstruct the composition range to keep it at the same
plain-text-character offsets against the new DOM nodes.

In this model, the JS editor would need to stash a copy of the previous DOM
at time of "beforeinput", allow the "input" to happen, and then reconstruct
the rich structure.  Then we could avoid the entire can of worms of what
particular IME events are "cancelable" or not and how to work around cases
where they are not.


--
Alex

On Mon, Feb 6, 2017 at 7:44 PM, Johannes Wilm <johannes@fiduswriter.org>
wrote:

>
>
> On Tue, Feb 7, 2017 at 4:00 AM, Alexandre Elias <aelias@google.com> wrote:
>
>> On Mon, Feb 6, 2017 at 6:18 PM, Johannes Wilm <johannes@fiduswriter.org>
>> wrote:
>>
>>>
>>>
>>> On Tue, Feb 7, 2017 at 12:23 AM, Alexandre Elias <aelias@google.com>
>>> wrote:
>>>
>>>> Hi, I'm the maintainer of IME on Android.  I'll be happy to attend the
>>>> conference call next week.  As you've surmised, Android does need to
>>>> perform a DOM change to merge two nodes "later".  However, I disagree that
>>>> those 5 steps are somehow needed.  Our plan is to send a single
>>>> insertCompositionText beforeinput event instead of 5 events.
>>>>
>>>
>>> Nice to meet you Alexandre!
>>>
>>> It's a bit unfrotunate that you haven't been able to participate in the
>>> last 3 years of discussions about this subject (I think some people have
>>> even been discussing this for more than 8 years), and it seems that the
>>> mailing list isn't as alive any more as it once used to be, but I'll do my
>>> best to give the reasoning why we came to the conclusion that these events
>>> are needed. I would also recommend consulting the various open and closed
>>> issues on github and to consult the archive of the mailing list. Also,
>>> various Google people have been behind a lot if not most of the various
>>> various concepts, so they can probably give you more background about this
>>> as well.
>>>
>>>
>>>>
>>>> Arguments why each is not needed:
>>>>
>>>>  > 1. deleteByComposition --This step is only needed when recomposing.
>>>> A composed word can go across element boundaries, so this may delete more
>>>> than just contents of a text node. (fx " He<b>ll</b><i>o fish</i>!"). This
>>>> step needs to be executed at the moment your Android IME changes the DOM,
>>>> whether this is at the moment the user taps the word and the underline
>>>> appears or at a later stage.
>>>>
>>>> There's no need for a separate "delete" event, because
>>>> insertion-to-replace-range is general enough to represent a deletion range
>>>> as well.  I prefer to reserve "delete" events exclusively for intended IME
>>>> deletions (e.g. select word and press backspace).
>>>>
>>>
>>> But I am a little confused: From the proposal Chong sent in, it sounded
>>> like you just wouldn't have any beforeinput event for this initial
>>> deletion. That would create a range of different problems, such as:
>>>
>>
>> I agree we should not hide any editing information from JS.  I am saying
>> that event 2/3 can encompass the deletion.  An implied, wrong assumption
>> underlying your argument seems to be that an insertion operation can only
>> ever add new text but not remove existing text.
>>
>
>
> No, that must be a misunderstanding. I don't know why you would think
> that. As I said previsouly, if  beforeinput events during IME are
> cancelable, 1 (deletion) and 2 (addition cshould be combinable into one
> single event,
>
>
> I don't quite get what you mean by combining steps two and three. Step two
> was the reinsertion of the initial composition string into the DOM. Step
> three is the change during the composition. There will usually be many step
> threes, but only one step two.
>
> The reason there is a step one and two at all is that the IME converts the
> composition string from being something that can contain a complex
> html-structure to just be  a plaintext string (in most cases anyway).
>
> So step 1 will remove "He<b>l</b>p" and step 2 will re-add it as "Help".
> Combining step 1 + 2 should work if step 2 is also cancelable. Combining
> step 2 and 3, or maybe 1 and 2 and 3 (?) would mean that we go straight
> from "He<b>l</b>p" to "Hel". and then the next two step 3 would add "l" and
> "o". In many cases the JS will probably be able to guess what the
> individual parts of this combination of 1/2/3 initially were (what part was
> just about turning it into a plaintext and which part was then about
> modifying the plaintext), but it would seem a bit meshed and I'm not sure
> one always can determine it. Nevertheless, if iME input is cancelable, then
> this does sound better than what was on the table previosuly.
>
>
>
>
>>   In reality, insertions are always *replacements* of the content of the
>> caret/selection/composition with text (for example, if you select text with
>> the mouse and then type "a" on a physical keyboard, no deletion event is
>> sent).
>>
>
> right. That'sw hy we have targetranges. But the difference for keyboard
> inout is that it is cancelable and there is no automatic normalization to
> plaintext. This is different than the situation with IME.
>
>>
>>
>>> * The more advanced editors today are moving toward a model where they
>>> hold a json version of the document which is separate from the DOM. They
>>> only make adjustments to the DOM when needed, and if the user changes
>>> something about the DOM, they make adjustments to the  json document. The
>>> json version is the authoritative copy of the document. For this to work,
>>> they need to know about any change to the DOM. If you make user-initiated
>>> DOM changes that are not causing any beforeinput event at all, the
>>> beforeinput event is somewhat useless, aand they will have to continue to
>>> monitor user initiated DOM-changes in other ways.
>>>
>>> * In a collaborative editor, if you make a change to the DOM of one
>>> user, and the JS is not being made aware of this change, then it won't
>>> notify the collaborators of this change, and so the collaborator will ahve
>>> different versions of the document. Again, the point of beforeinput events
>>> will be gone. JS develoeprs will use mutation observers or some other
>>> technology instead.
>>>
>>>
>>>
>>>
>>>>
>>>> > 2. insertCompositionText -- Also this step only needs to happen at
>>>> the moment only for recompositions. It reinserts the word from step one,
>>>> but this time as plaintext in one element (fx "Hello" => "<i>Hello
>>>> fish</i>")
>>>>
>>>> On Android because of the "late" timing, this merge operation would
>>>> happen in the same message loop task as step 3, so there is no need for
>>>> this event.  We prefer to go directly from "He<b>ll</b><i>o fish</i>" ->
>>>> "<i>Help<i>".
>>>>
>>>
>>>
>>> Yes, in a perfect world, where all IME-related events are cancelable,
>>> this could be done and would be preferable, as for example Yosin has
>>> pointed out ( https://github.com/w3c/input-e
>>> vents/issues/45#issuecomment-261427648 ). We had a long discussion
>>> during the fall of 2015 about this, and among other ones several developers
>>> from Chromium eventually convinced us here that IME-events cannot be
>>> guaranteed to be cancelable. Even moving the caret to somewhere else during
>>> the composition may somehow crash the IME (hwo this is possible is a bit of
>>> a mystery to me as a JS developer).
>>>
>>> One could then say ok, combine 1 and 2 and make them both
>>> non-cancelable. The problem with that is that the browser doesn't always
>>> know the seamntic meaning of the various DOM elements used within a
>>> document and it may end up making decisions of what to delete that are
>>> problematic.
>>>
>>
>> I've been trying to figure out where these claims about IME from some
>> Chromium team members came from exactly and haven't figured it out, nobody
>> has a precise memory.  They seem to be based on a vague fears rather than
>> concrete knowledge of how IMEs behave.  I agree with your instinct that the
>> assumptions behind it don't make too much sense, that's why I think we
>> should hold off before we commit to a complicated model based on those
>> assumptions.
>>
>
> Well, maybe you guys need to discuss that internally. Until now what we
> said here is that we will assume the worst case scenario (that IMEs not
> being allowed to directly modify the DOM will lead to them crashing) and
> then in a future version to hopefully get one that doesn't crash.
>
> Btw -- just to make it clear because this seems to have been a common
> misconception between JS developers and browser developers in the past:
> When we ask for these things to be cancelable, in 80% of cases this is not
> because we want to stop the user from doing things. It's more than we want
> to do the corresponding DOM changes in JS code, because they are specific
> to the JS editor the user is currently using and they may not necessarily
> make sense to the browser.
>
>
>
>>
>>
>>>
>>>
>>>
>>>> However, rniwa@apple.com has told us that on OS X, the standard
>>>> platform behavior is to merge the nodes at time of composition start.
>>>> Therefore, on OS X step 2 is still needed and there would be 2 beforeinput
>>>> events in total, but on Android there would be only 1.  This represents a
>>>> substantive platform difference that I think it makes sense to allow to
>>>> differ in the spec.  I don't think we should try to artificially
>>>> synchronize the platforms by always sending 2 events -- on Android this
>>>> would simply result in unnecessary work and side effects that would never
>>>> be seen by the user.
>>>>
>>>
>>> I think you should instead consider making this two events on Android.
>>> Android is really great, but text editing in Chrome on Android is really
>>> problematic especially when it comes to IME recomposition, as I am sure you
>>> are aware. I have tried various keyboard, and but even on mainstream sites
>>> such as Facebook, using a Google-provided keyboard, recomposition tends to
>>> just mess up entire sentences, throw random words around, deleting or
>>> merging words that are close to what one is trying to recompose, etc.
>>>
>>
>> Right, we are in process of fixing all of these bugs and expect to get
>> the remaining ones fixed in M58/M59 timeframe.  Anyway, this doesn't
>> address my point.  Personally, I believe that firing additional unnecessary
>> events will prove harmful to both UAs and JS editors -- in reality an
>> atomic IME change that just happened, but you are proposing to drip-feed it
>> into many separate events.  Your proposal doesn't provide JS any additional
>> information (it just splits information into pieces), so therefore I don't
>> think it can possibly be needed for any use case either.
>>
>>
>>
>>> Speed is not as much a concern anyway because it all depends on the
>>> human who is interacting with it, but in addition these really major issues
>>> mean that recomposition as it exists on Android as of today is not very
>>> useful. In addition there are the various issues that JS developers have
>>> complained about for a while: no compositionstart event, etc. . I am sure
>>> you know the list of grievances with it all too well.
>>>
>>
>> We should always be firing compositionStart today.  Feel free to file a
>> bug against me if you know of a case where we don't.
>>
>
> I recently sent a list of prioritized bugs from several JS editor projects
> to the Chromium team that they had requested from em. I assume they
> forwarded those issues relevant to Android to you.
>
>>
>>
>>>
>>> Not knowing the details, but I could imagine you can check for event
>>> listeners, and only if those are present you make it two events.
>>>
>>> But as an alternative, and to try to get a bit closer to both your and
>>> Yosin's position, how about this:  We make two IME models, one for IMEs
>>> where the beforeinput events can be cancelled in which step 1 and 2 are
>>> merged and one where they cannot be cancelled where they are separate.
>>>
>>
>> Disagree, we should keep things as simple as possible.  I want us to fire
>> a minimum of events and avoid complicated logic forking.  I don't want us
>> to introduce permanent complexity into the platform for the sake of a
>> short-term compromise.
>>
>
> Well, you said that Apple does one thing and you do another. So either the
> two of you can agree or you don't. If you cannot agree, we'll have to have
> two different models.
>
> In theis scenario, if the Chrome model consists of just cutting out 4 of 5
> events on Android, then as I tried to explain in the last few emails and
> the comments on the proposal doc, the event will no longer function to
> build an editor on top of. At least not without combining it with mutation
> observers and other technologies.
>
> Having a model where IME events are cancelable is really something we all
> hoped to do in a future version of the spec. If the situation is such that
> Chromium/Android would prefer to do this right now, whereas Apple prefers
> to stay with the non-cancelable IME-model then that would not seem like
> terribly different models.
>
>
>>
>>
>>>
>>>>
>>> > 4. deleteCompositionText -- removed (fx "help"): It removed the text
>>>> from the underline and it may not be the entire composition.
>>>>
>>>> > 5. insertFromComposition -- adds "help" permanently, and gives the JS
>>>> the opposrtunity to rebuild the complex HTML structure lost in step 1.
>>>>
>>>> Step 3 is already permanent.  The fact that there is still an active
>>>> composition underline at the time of step 3 does not make it somehow
>>>> temporary.  It has already been injected into the DOM and caused all side
>>>> effects.  So the JS should rebuild its complex structure at time of event
>>>> 3, and events 4/5 are redundant.
>>>>
>>>
>>> When a composition starts, they temporarily pause most activities
>>> because they are not really allowed to intervene. Then, once the
>>> composition is finished, they deal with the finished text.  input. For
>>> example, in step 5 they may reapply some partial styling that was lost in
>>> step 1.
>>>
>>
>> "compositionend" event is fired at the same timing that steps 4/5 would
>> be, isn't it sufficient for this use case?
>>
>
> There is one problem in that compositionend with data="" could mean the
> composition has been cancelled or that the final compositionstring is
> empty. The distinction between 4 and 5 is needed if 2, 3 & 4 are
> non-cancelable. If they are cancelable, then one should be able to combine
> 1 and 2, and skip 4 and 5 and the JS would continously make any adjustments
> it needs to the DOM (while taking into consideration what the
> compositionstring is at any moment).
>
>
>>
>>
>>>
>>>
>>>
>>>>
>>>>>
>>>>> The reason you state for the proposed change is that "On Android it’s
>>>>> very common for IME to start composition whenever a cursor is moved onto a
>>>>> word.  This is primarily intended to passively show an underline to the
>>>>> user (to indicate what would be replaced *if* user selects an alternate
>>>>> spelling)."
>>>>>
>>>>> If such tapping leads to actual DOM changes, then  the JS needs to
>>>>> know about those(step 1/2). But if you don't change the DOM immediately and
>>>>> only do so at a later stage when you "really" start the composition, then
>>>>> you can also wait until that stage.
>>>>>
>>>>>
>>>>> So these steps are needed on allplattforms, no matter whether or not
>>>>> you change how IMEs work on Android.
>>>>>
>>>>>
>>>>>>
>>>>>> Adding aelias@ to make sure he is able to make the call on the 14th.
>>>>>>
>>>>>> dave.
>>>>>>
>>>>>> On Fri, Feb 3, 2017 at 1:33 AM, Johannes Wilm <
>>>>>> johannes@fiduswriter.org> wrote:
>>>>>>
>>>>>>> Could we get the person or people who came up with these changes due
>>>>>>> to Android to participate in the call on the 14th? I think it's very
>>>>>>> important to have them invovled in this discussion.
>>>>>>>
>>>>>>> On Fri, Feb 3, 2017 at 10:21 AM, Johannes Wilm <
>>>>>>> johannes@fiduswriter.org> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 3, 2017 at 10:18 AM, Johannes Wilm <
>>>>>>>> johannes@fiduswriter.org> wrote:
>>>>>>>> ...
>>>>>>>>
>>>>>>>>> Right now I cannot see any purpose of the beforeinput event with
>>>>>>>>> these changes applied, but maybe I am missing something?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I need to restarct that. I guess it could still be useful to stop
>>>>>>>> native bold/italic buttons from making their own, non-controlled DOM
>>>>>>>> changes. It's just for text input that it is irrelevant. And IME will
>>>>>>>> continue to be a mess.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Johannes Wilm
>>>>>>>> Fidus Writer
>>>>>>>> http://www.fiduswriter.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Johannes Wilm
>>>>>>> Fidus Writer
>>>>>>> http://www.fiduswriter.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Johannes Wilm
>>>>> Fidus Writer
>>>>> http://www.fiduswriter.org
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Johannes Wilm
>>> Fidus Writer
>>> http://www.fiduswriter.org
>>>
>>
>>
>
>
> --
> Johannes Wilm
> Fidus Writer
> http://www.fiduswriter.org
>