Re: explanatory note: preventDefault, IME & keyboard input

On Sat, Oct 17, 2015 at 12:22 PM, Koji Ishii <kojiishi@gmail.com> wrote:

>
>
> On Sat, Oct 17, 2015 at 7:46 AM, Johannes Wilm <johannes@fiduswriter.org>
> wrote:
>
>>
>>
>> On Fri, Oct 16, 2015 at 8:27 PM, Koji Ishii <kojiishi@gmail.com> wrote:
>>
>>> What I'd recommend for your problem is that you should simply let IME
>>> inserts composition string and ignore that, because by definition (at least
>>> theoretically) all composition strings are either to be removed or
>>> committed.
>>>
>>> When committed, it's a normal text typing operation. You can
>>> preventDefault, create <ins>, and re-insert.
>>>
>>
>>> It is the normal IME operation flow to remove composition string on
>>> commits, but maybe browsers may not tell JS that composition strings are
>>> removed and committed strings are inserted, because browsers (and probably
>>> most apps that do inline composition operations) may optimize it by just
>>> changing attributes of composition strings.
>>>
>>
>> Ok, you seem to be referring to some of the internal handling of IME
>> within the browser, correct? Now for us to have access to any of this, we
>> need to have an event at the right place, right?
>>
>
> Not sure if understand exactly what you wrote, but probably we're saying
> different things IIUC. This point is hard to discuss, because we have
> different approach to an issue, but to discuss the approach, we need
> detailed examples. Thank you for trying to understand.
>
> When things are not interoperable, there are two high-level approaches:
>
> 1. Make everything detectable, cancellable, modifiable, and allow
> upper-layers to fix things.
> 2. Abstract non-interoperable portion, make them invisible, but make sure
> its surface is interoperable.
>
> I think you're trying approach #1, while I recommend approach 2, because
> those "everything" in approach 1 is hard to make interoperable across
> platforms.
>
> When something troubles JS, your approach is to let JS to cancel whatever
> undesired within the action stream. My approach is to build an abstracted,
> documented, and interoperable higher-level interface that can hide what
> troubles JS. When Ryosuke used "abstract", I think he meant closer to my
> approach.
>
> Do you see the differences in these approaches? I'm not expecting us to
> reach consensus in one day, but having you understand, whenever I suggest
> something, it's based on approach 2, would probably help our discussions a
> lot.
>

I see the difference in approach. And as we have JS people have said many
times, we are open to alternative solutions that solve the same problems.

However, we then also need the alternative solution, not just being against
everything.

As for the two approaches, they seem to be translateable into:

1. JS does most things
2. browser does most things

I would be fine with either one, but browsers do not seem to have the
budgets or interest to employ say 50 more fulltime staff each to take over
thew job of building all the standard stuff of editors into their programs.
So that limits approach 2 and this editing taskforce has been working on
making more of the primitives available to make life for JS editor
developers easier.

Still one can try to make 2 work for some things. We had initially
suggested to have IME character composition happen transparently in a
Shadow DOM-alike entity and then only get a beforeInput event for the
finished string. Ojan Vafai made clear that is a no-go for Chrome due to
performance issues, so we have to find other ways. Ojan proposed moving the
caret at compositionstart, which is possible and legal per spec as of
today. You are against doing this. Could you then provide an alternative
that let's us achieve the same goal, keeping in mind that it is JS editors
that have to govern what goes into the portion of the DOM that their editor
is controlling, and not the browser?

At the F2F we talked about how eventually the entire cE=true could be
specced (within some decades or so). This is a task that no-one is starting
on now and in this taskforce. But we do have the spec which we are just
maintaining as little , and if you want to become the editor of that spec,
I am sure we can make you be that.


> From the specification I can see that the compositionstart event can be
>> canceled, while the compositionend event cannot. Is this where you think I
>> could enter a preventDefault?
>>
>
> No, that's approach 1. I'm negative to make any composition events
> cancelable, including compositionstart.
>

So you are proposing to take away what is legal today. It seems like that
is more about restricting the options of JS editor developers, not helping
them. Could you then give an alternative for us people on the JS side of
things to achieve the same?




>
>> The inserted text in text tracking is normally marked with a specific
>> color. If we moved the carets into an "<ins>"-element after the
>> compositionend event, the text would initially appear in the default text
>> color and only when the word has been finished would it change the color to
>> match the other inserted text. That would be very unfamiliar to the end
>> user, who is used to write the added text that is being tracked immediately
>> in a color that marks his tracked changes. That is why moving the caret at
>> the time of compositionstart to a newly created place in the DOM is needed.
>>
>
> I understand that, but such text has specific visual to indicate it's a
> composition text. Sometimes with underlines, some systems change
> text/background color, so apps can't control every aspects of the visuals.
> Some IME even allows users to change visuals of composition strings. Better
> if we can make it closer within the restriction, but as long as committed
> text matches to what you want, I think we can defer that to future levels.
>
>
In both Android for European languages and the IMEs on Chrome for Japanese
that I have been trying out, it seems to take the color and the font-weight
of the element that surroundings the text that is to be edited. This seems
to cover quite a few of the users already. Do we have percentages of how
many users will lose font color and text weight information during IME
composition? Is there any way to find out about this?


> Composition strings are always temporary, it'd never be saved to documents.
>

With "composition string" you mean the string that is currently being
constructed within the IME (the characters that have a line underneath
them). Whenever such a character is added, at least in Chrome the DOM is
updated, so if the JS editor decides to save the document during
composition, the half-composed character will be saved to the document, no
matter what, right?


For that, system does all the efforts to make sure composition strings are
> committed on several events. I wish you agree on the priority of visuals of
> the composition strings in specific cases, if not, well, we've got find
> something yet another solution. I do have some ideas, but it's not the
> direction to intervene IME events streams, and it could be a bit complex
> that I'd prefer to defer.
>

As stated before, the most important thing is to be able to move the caret
during compositionstart so one can sandbox the entire thing. Changing the
string that is currently being constructed during construction would have
to be something related to advanced character replacement. It is only if
you take away the possibility to sandbox, that these other things become
really necessary.


>
> The preventDefault default I was proposing was for each of the character
>> insertion or character deletion events during composition, not at
>> compositionstart. If I understand you corect it is very
>> difficult/impossible to do preventDefault at compositionstart. Maybe so,
>> but currently according to the spec that should be a feature of
>> compositionstart already.
>>
>
> I don't know which spec mentions that, sorry about that, but sounds like
> the spec should be fixed then. I assume it's not REC nor implemented
> interoperably yet, if so, that explains it's not really implementable
> across platforms nor have enough attentions yet.
>


The UI events spec.
http://www.w3.org/TR/DOM-Level-3-Events/#event-type-compositionstart .


Can you explain why you need to take that away? If it is currently
possible, then it can't really be for technical reasons. And you can stop
input from the keyboard as the situation is now, so why would one be
disallowed to stop input coming from an IME?



>
> The preventDefault for IME-based insert/delete character actions was
>> originally proposed as an alternative in case it was impossible to move the
>> caret during compositionstart/compositionend. At the same time, it would us
>> to exchange characters according to complex rules, etc. . FOr ICE we had to
>> do our own space insertion to replace the normal space insertion with a
>> scientific space, for example.
>>
>
> Since that was a mistake that should be fixed in my personal opinion, I'm
> against putting more into the bucket.
>
> I think we can make sure this optimization is disabled if event listeners
>>> exist (or opted-in by a new function), and give JS an opportunity to handle
>>> committed text just the same way as inputs from hardware keyboard.
>>>
>>>
>> Does this solve your problem?
>>>
>>> Ok, you seem to be referring to some of the internal handling of IME
>>> within the browser, correct? Now for us to have access to any of this, we
>>> need to have an event at the right place, right?
>>>
>>> From the specification I can see that the compositionstart event can be
>>> canceled, while the compositionend event cannot. Is this where you think I
>>> could enter a preventDefault?
>>>
>>> The inserted text in text tracking is normally marked with a specific
>>> color. If we moved the carets into an "<ins>"-element after the
>>> compositionend event, the text would initially appear in the default text
>>> color and only when the word has been finished would it change the color to
>>> match the other inserted text. That would be very unfamiliar to the end
>>> user, who is used to write the added text that is being tracked immediately
>>> in a color that marks his tracked changes. That is why moving the caret at
>>> the time of compositionstart to a newly created place in the DOM is needed.
>>>
>>> The preventDefault default I was proposing was for each of the character
>>> insertion or character deletion events during composition, not at
>>> compositionstart. If I understand you corect it is very
>>> difficult/impossible to do preventDefault at compositionstart. Maybe so,
>>> but currently according to the spec that should be a feature of
>>> compositionstart already.
>>>
>>> The preventDefault for IME-based insert/delete character actions was
>>> originally proposed as an alternative in case it was impossible to move the
>>> caret during compositionstart/compositionend. At the same time, it would us
>>> to exchange characters according to complex rules, etc. . FOr ICE we had to
>>> do our own space insertion to replace the normal space insertion with a
>>> scientific space, for example.
>>>
>>
>> When programming the polyfill for doing IME composition in the ShadowDOM,
>> I noticed that if I move the caret back to where the caret was when I
>> started the composition, it would then insert all the constructed
>> characters in the place where the caret is moved to during compositionend
>> in Chrome (not Firefox). Is that possibly what you are referring to?
>>
>> If we could put the behavior of Chrome on into a spec, it would indeed
>> help when creating new characters.
>>
>
> Ah...I'm a bit lost where this came from in the context of this
> discussion, but I remember you proposed that. If you're referring to that
> issue, no, my opinion is do not intervene composition strings, but
> abstract, and make it possible to process after committed. Don't move
> carets during compositions. If it's really needed to be defined, I'd vote
> to commit when caret/selection moves.
>

You wrote "It is the normal IME operation flow to remove composition string
on commits, but maybe browsers may not tell JS that composition strings are
removed and committed strings are inserted, because browsers (and probably
most apps that do inline composition operations) may optimize it by just
changing attributes of composition strings."

I guess that with "commit" you mean "compositionend" and with "composition
string" you mean the text that has been contructed within the IME. If they
are other entitites, then it sems like they are things we cannot get in
contact with on the JS side of things, right?

So assuming this is what you mean, I read you as saying that "when the
composition has ended, we first delete the characters that have been
created from the DOM, and then we reinsert them again. And you can
preventDefault this insertion, etc.".

So my answer:

- The compositionend cannot currently be canceled, so are you proposing
making it be canceable?
- When writing my polyfill and moving the caret somewhere else during
compositionend, I noticed that on Chrome I get the composed characters
inserted at the place where the caret has been moved to. This is possibly
the re-insertion you were talking about? If so, I am for having this happen
on all browsers.

You then answer that you are against that. May I ask what alternative you
propose?


>
> However, this does not cover the usecase of Android/iOS or other IMEs that
>> modify existing content. These are the issues I see for that:
>>
>> 1. DOM structure around the word in question is being destroyed and
>> replaced with one the browser assumes is OK, which will place the entire
>> word into one single element. There needs to be a way to stop this, or at
>> least reverse it after composition has ended without having to resort to
>> complex clientside diffing libraries.
>>
>> 2. the event.data gives us the word the browser thinks it recognized as a
>> simple text string such as "Aple". But it says nothing about what range
>> that corresponds to. Trying to find the word and the range it corresponds
>> to across different elements in JS alone is a huge task. It would help a
>> lot if we got access to this info directly. This should also allow us to
>> sandbox the entire currently edited word.
>>
>> 3. it would help even more if we had a way to change what the browser
>> sees as the relevant word at the time of compositionstart.
>>
>> 4. Also here, sandboxing it will help a lot, both to cut down on IME
>> weirdness and differences, but also in order to hold the DOm clean from
>> characters that will not make it into the final version.
>>
>
> Yeah, re-conversion (is this old term? not sure...) requires additional
> work for sure. I haven't made all the necessary thinking yet, but my
> approach stays to abstract and build an interoperable layer on top of
> platform-specific layers, rather than standardizing all the IME details.
>
> If we can agree on the overall approach, I think we can discuss more on
> the re-conversion.
>

The overall approach has been given to this taskforce, and unless the
browser makers change their mind and invest in a large amount of fulltime
staff just for editing, I don't think there is any chance of changing it.

But when it comes to IME specifically, if you can come up with a proposal
that solves it in the browser in a way that lets us achieve the problems
that we JS people have explained over the past 1.5 years, then I don't
think any of us are against it. But you need to concretely address these
problems. Just insisting the browser will handle it, but bug reports not
being fixed for years and no concrete proposals of how browsers would do
it, doesn't cut it.


-- 
Johannes Wilm
Fidus Writer
http://www.fiduswriter.org