Re: explanatory note: preventDefault, IME & keyboard input

On Fri, Oct 16, 2015 at 8:27 PM, Koji Ishii <kojiishi@gmail.com> wrote:

> What I'd recommend for your problem is that you should simply let IME
> inserts composition string and ignore that, because by definition (at least
> theoretically) all composition strings are either to be removed or
> committed.
>
> When committed, it's a normal text typing operation. You can
> preventDefault, create <ins>, and re-insert.
>

> It is the normal IME operation flow to remove composition string on
> commits, but maybe browsers may not tell JS that composition strings are
> removed and committed strings are inserted, because browsers (and probably
> most apps that do inline composition operations) may optimize it by just
> changing attributes of composition strings.
>



Ok, you seem to be referring to some of the internal handling of IME within
the browser, correct? Now for us to have access to any of this, we need to
have an event at the right place, right?

>From the specification I can see that the compositionstart event can be
canceled, while the compositionend event cannot. Is this where you think I
could enter a preventDefault?

The inserted text in text tracking is normally marked with a specific
color. If we moved the carets into an "<ins>"-element after the
compositionend event, the text would initially appear in the default text
color and only when the word has been finished would it change the color to
match the other inserted text. That would be very unfamiliar to the end
user, who is used to write the added text that is being tracked immediately
in a color that marks his tracked changes. That is why moving the caret at
the time of compositionstart to a newly created place in the DOM is needed.

The preventDefault default I was proposing was for each of the character
insertion or character deletion events during composition, not at
compositionstart. If I understand you corect it is very
difficult/impossible to do preventDefault at compositionstart. Maybe so,
but currently according to the spec that should be a feature of
compositionstart already.

The preventDefault for IME-based insert/delete character actions was
originally proposed as an alternative in case it was impossible to move the
caret during compositionstart/compositionend. At the same time, it would us
to exchange characters according to complex rules, etc. . FOr ICE we had to
do our own space insertion to replace the normal space insertion with a
scientific space, for example.


>
> I think we can make sure this optimization is disabled if event listeners
> exist (or opted-in by a new function), and give JS an opportunity to handle
> committed text just the same way as inputs from hardware keyboard.
>
>
Does this solve your problem?
>
> Ok, you seem to be referring to some of the internal handling of IME
> within the browser, correct? Now for us to have access to any of this, we
> need to have an event at the right place, right?
>
> From the specification I can see that the compositionstart event can be
> canceled, while the compositionend event cannot. Is this where you think I
> could enter a preventDefault?
>
> The inserted text in text tracking is normally marked with a specific
> color. If we moved the carets into an "<ins>"-element after the
> compositionend event, the text would initially appear in the default text
> color and only when the word has been finished would it change the color to
> match the other inserted text. That would be very unfamiliar to the end
> user, who is used to write the added text that is being tracked immediately
> in a color that marks his tracked changes. That is why moving the caret at
> the time of compositionstart to a newly created place in the DOM is needed.
>
> The preventDefault default I was proposing was for each of the character
> insertion or character deletion events during composition, not at
> compositionstart. If I understand you corect it is very
> difficult/impossible to do preventDefault at compositionstart. Maybe so,
> but currently according to the spec that should be a feature of
> compositionstart already.
>
> The preventDefault for IME-based insert/delete character actions was
> originally proposed as an alternative in case it was impossible to move the
> caret during compositionstart/compositionend. At the same time, it would us
> to exchange characters according to complex rules, etc. . FOr ICE we had to
> do our own space insertion to replace the normal space insertion with a
> scientific space, for example.
>



When programming the polyfill for doing IME composition in the ShadowDOM, I
noticed that if I move the caret back to where the caret was when I started
the composition, it would then insert all the constructed characters in the
place where the caret is moved to during compositionend in Chrome (not
Firefox). Is that possibly what you are referring to?

If we could put the behavior of Chrome on into a spec, it would indeed help
when creating new characters.



However, this does not cover the usecase of Android/iOS or other IMEs that
modify existing content. These are the issues I see for that:

1. DOM structure around the word in question is being destroyed and
replaced with one the browser assumes is OK, which will place the entire
word into one single element. There needs to be a way to stop this, or at
least reverse it after composition has ended without having to resort to
complex clientside diffing libraries.

2. the event.data gives us the word the browser thinks it recognized as a
simple text string such as "Aple". But it says nothing about what range
that corresponds to. Trying to find the word and the range it corresponds
to across different elements in JS alone is a huge task. It would help a
lot if we got access to this info directly. This should also allow us to
sandbox the entire currently edited word.

3. it would help even more if we had a way to change what the browser sees
as the relevant word at the time of compositionstart.

4. Also here, sandboxing it will help a lot, both to cut down on IME
weirdness and differences, but also in order to hold the DOm clean from
characters that will not make it into the final version.





> /koji
>
> On Fri, Oct 16, 2015 at 4:09 PM, Johannes Wilm <johannes@fiduswriter.org>
> wrote:
>
>> Hey,
>> there seem to recently been have some confusion about IME and whether
>> allowing preventDefault on IME-based beforeInput events would mainly be
>> used to disable IME-input.
>>
>> *What is the current situation?*
>>
>> Let me exemplify with ICE -- the track changes plugin that is
>> contentEditable based [1]. I contributed a great deal on getting it to work
>> on Chrome a few years ago. It listens to keypress/keyup/keydown events and
>> preventDefaults these under certain circumstances [2]. But why?
>>
>> Say you have a normal non-tracked text:
>>
>> "Hello this is a te|t text"
>>
>> | = caret
>>
>>
>> There is an 's' missing where the caret is, and the user wants to insert
>> it. If we don't preventDefault anything, it will turn into:
>>
>> "Hello, this is a test text"
>>
>> but because we track changes, we need to turn it into:
>>
>> "Hello, this is a te<ins>s</ins>t text." (simplified)
>>
>> We can't really create the <ins>-element before the user actually hits a
>> key, because unless the user actually enters a letter, we cannot just add a
>> random element there just because the caret is there.
>>
>>
>> *What will the situation look like with our spec?*
>>
>> Once our beforeEdit/beforeInput command is available, for keyboard input
>> we will listen to that instead of the keypress event. The above example
>> will therefore work almost the same.
>>
>>
>> How about if the s came from IME-input? The current state of negotiations
>> is that the beforeInput event can be cancelled for every event EXCEPT those
>> based on IME-input. Additionally, the discussion in the uievents thread
>> show that we are not even allowed to sandbox IME input.
>>
>>
>> The problem at hand would be that it would immediately enter the s
>> without us being able to stop it:
>>
>> "Hello, this is a test text"
>>
>> We would then need to have saved a previous version of the DOM and a diff
>> finding function on the two versions, find out that the difference is the
>> "s" and then create an <ins>-element in it's place and exchange the current
>> text for:
>>
>> "Hello, this is a te<ins>s</ins>t text."
>>
>>
>> And this is just the first problem. The way Android/iOS handle what they
>> see as "words" probably means that they will randomly remove <ins>-elements
>> entirely or put entire words inside of <ins>-elements.
>>
>> *So what am I proposing?*
>>
>> I am proposing that also IME-based beforeInput events should be
>> preventDefaultable, as a first step to making it easier to handle them.
>>
>> Additionally, if IMEs work on more than just simple text nodes, we
>> probably both need a way to tell the browser what it should consider an
>> entire word, and preferably also a way to sandbox them (which btw is really
>> easy for keyboard-based input).
>>
>> I recognize Ryosuke's answer that the JS then also need to tell the
>> browser after a browser handled text insertion what range it should
>> consider as having been input by the user in order to both
>>
>> a) Continue IME-composition , underlining everything correctly.
>>
>> b) Let the browser know which parts of the text are still in need of
>> spell checking/autofixing and which parts should be considered as having
>> been edited by the user.
>>
>>
>>
>> ------
>>
>> Did that make it clearer?
>>
>> ------
>>
>> [1] https://github.com/NYTimes/ice/
>> [2] https://github.com/NYTimes/ice/blob/master/src/ice.js#L1633
>> --
>> Johannes Wilm
>> Fidus Writer
>> http://www.fiduswriter.org
>>
>
>


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org