Re: explanatory note: preventDefault, IME & keyboard input

On Sat, Oct 17, 2015 at 7:46 AM, Johannes Wilm <johannes@fiduswriter.org>
wrote:

>
>
> On Fri, Oct 16, 2015 at 8:27 PM, Koji Ishii <kojiishi@gmail.com> wrote:
>
>> What I'd recommend for your problem is that you should simply let IME
>> inserts composition string and ignore that, because by definition (at least
>> theoretically) all composition strings are either to be removed or
>> committed.
>>
>> When committed, it's a normal text typing operation. You can
>> preventDefault, create <ins>, and re-insert.
>>
>
>> It is the normal IME operation flow to remove composition string on
>> commits, but maybe browsers may not tell JS that composition strings are
>> removed and committed strings are inserted, because browsers (and probably
>> most apps that do inline composition operations) may optimize it by just
>> changing attributes of composition strings.
>>
>
> Ok, you seem to be referring to some of the internal handling of IME
> within the browser, correct? Now for us to have access to any of this, we
> need to have an event at the right place, right?
>

Not sure if understand exactly what you wrote, but probably we're saying
different things IIUC. This point is hard to discuss, because we have
different approach to an issue, but to discuss the approach, we need
detailed examples. Thank you for trying to understand.

When things are not interoperable, there are two high-level approaches:

1. Make everything detectable, cancellable, modifiable, and allow
upper-layers to fix things.
2. Abstract non-interoperable portion, make them invisible, but make sure
its surface is interoperable.

I think you're trying approach #1, while I recommend approach 2, because
those "everything" in approach 1 is hard to make interoperable across
platforms.

When something troubles JS, your approach is to let JS to cancel whatever
undesired within the action stream. My approach is to build an abstracted,
documented, and interoperable higher-level interface that can hide what
troubles JS. When Ryosuke used "abstract", I think he meant closer to my
approach.

Do you see the differences in these approaches? I'm not expecting us to
reach consensus in one day, but having you understand, whenever I suggest
something, it's based on approach 2, would probably help our discussions a
lot.

>From the specification I can see that the compositionstart event can be
> canceled, while the compositionend event cannot. Is this where you think I
> could enter a preventDefault?
>

No, that's approach 1. I'm negative to make any composition events
cancelable, including compositionstart.


> The inserted text in text tracking is normally marked with a specific
> color. If we moved the carets into an "<ins>"-element after the
> compositionend event, the text would initially appear in the default text
> color and only when the word has been finished would it change the color to
> match the other inserted text. That would be very unfamiliar to the end
> user, who is used to write the added text that is being tracked immediately
> in a color that marks his tracked changes. That is why moving the caret at
> the time of compositionstart to a newly created place in the DOM is needed.
>

I understand that, but such text has specific visual to indicate it's a
composition text. Sometimes with underlines, some systems change
text/background color, so apps can't control every aspects of the visuals.
Some IME even allows users to change visuals of composition strings. Better
if we can make it closer within the restriction, but as long as committed
text matches to what you want, I think we can defer that to future levels.

Composition strings are always temporary, it'd never be saved to documents.
For that, system does all the efforts to make sure composition strings are
committed on several events. I wish you agree on the priority of visuals of
the composition strings in specific cases, if not, well, we've got find
something yet another solution. I do have some ideas, but it's not the
direction to intervene IME events streams, and it could be a bit complex
that I'd prefer to defer.

The preventDefault default I was proposing was for each of the character
> insertion or character deletion events during composition, not at
> compositionstart. If I understand you corect it is very
> difficult/impossible to do preventDefault at compositionstart. Maybe so,
> but currently according to the spec that should be a feature of
> compositionstart already.
>

I don't know which spec mentions that, sorry about that, but sounds like
the spec should be fixed then. I assume it's not REC nor implemented
interoperably yet, if so, that explains it's not really implementable
across platforms nor have enough attentions yet.

The preventDefault for IME-based insert/delete character actions was
> originally proposed as an alternative in case it was impossible to move the
> caret during compositionstart/compositionend. At the same time, it would us
> to exchange characters according to complex rules, etc. . FOr ICE we had to
> do our own space insertion to replace the normal space insertion with a
> scientific space, for example.
>

Since that was a mistake that should be fixed in my personal opinion, I'm
against putting more into the bucket.

I think we can make sure this optimization is disabled if event listeners
>> exist (or opted-in by a new function), and give JS an opportunity to handle
>> committed text just the same way as inputs from hardware keyboard.
>>
>>
> Does this solve your problem?
>>
>> Ok, you seem to be referring to some of the internal handling of IME
>> within the browser, correct? Now for us to have access to any of this, we
>> need to have an event at the right place, right?
>>
>> From the specification I can see that the compositionstart event can be
>> canceled, while the compositionend event cannot. Is this where you think I
>> could enter a preventDefault?
>>
>> The inserted text in text tracking is normally marked with a specific
>> color. If we moved the carets into an "<ins>"-element after the
>> compositionend event, the text would initially appear in the default text
>> color and only when the word has been finished would it change the color to
>> match the other inserted text. That would be very unfamiliar to the end
>> user, who is used to write the added text that is being tracked immediately
>> in a color that marks his tracked changes. That is why moving the caret at
>> the time of compositionstart to a newly created place in the DOM is needed.
>>
>> The preventDefault default I was proposing was for each of the character
>> insertion or character deletion events during composition, not at
>> compositionstart. If I understand you corect it is very
>> difficult/impossible to do preventDefault at compositionstart. Maybe so,
>> but currently according to the spec that should be a feature of
>> compositionstart already.
>>
>> The preventDefault for IME-based insert/delete character actions was
>> originally proposed as an alternative in case it was impossible to move the
>> caret during compositionstart/compositionend. At the same time, it would us
>> to exchange characters according to complex rules, etc. . FOr ICE we had to
>> do our own space insertion to replace the normal space insertion with a
>> scientific space, for example.
>>
>
> When programming the polyfill for doing IME composition in the ShadowDOM,
> I noticed that if I move the caret back to where the caret was when I
> started the composition, it would then insert all the constructed
> characters in the place where the caret is moved to during compositionend
> in Chrome (not Firefox). Is that possibly what you are referring to?
>
> If we could put the behavior of Chrome on into a spec, it would indeed
> help when creating new characters.
>

Ah...I'm a bit lost where this came from in the context of this discussion,
but I remember you proposed that. If you're referring to that issue, no, my
opinion is do not intervene composition strings, but abstract, and make it
possible to process after committed. Don't move carets during compositions.
If it's really needed to be defined, I'd vote to commit when
caret/selection moves.

However, this does not cover the usecase of Android/iOS or other IMEs that
> modify existing content. These are the issues I see for that:
>
> 1. DOM structure around the word in question is being destroyed and
> replaced with one the browser assumes is OK, which will place the entire
> word into one single element. There needs to be a way to stop this, or at
> least reverse it after composition has ended without having to resort to
> complex clientside diffing libraries.
>
> 2. the event.data gives us the word the browser thinks it recognized as a
> simple text string such as "Aple". But it says nothing about what range
> that corresponds to. Trying to find the word and the range it corresponds
> to across different elements in JS alone is a huge task. It would help a
> lot if we got access to this info directly. This should also allow us to
> sandbox the entire currently edited word.
>
> 3. it would help even more if we had a way to change what the browser sees
> as the relevant word at the time of compositionstart.
>
> 4. Also here, sandboxing it will help a lot, both to cut down on IME
> weirdness and differences, but also in order to hold the DOm clean from
> characters that will not make it into the final version.
>

Yeah, re-conversion (is this old term? not sure...) requires additional
work for sure. I haven't made all the necessary thinking yet, but my
approach stays to abstract and build an interoperable layer on top of
platform-specific layers, rather than standardizing all the IME details.

If we can agree on the overall approach, I think we can discuss more on the
re-conversion.

/koji