Re: [w3c/editing] Seeking feedback on delayed clipboard rendering proposal (Issue #417)

Discussed at editing meeting 3/9/2023
[08:02:27]  <GameMaker> Issue: https://github.com/w3c/editing/issues/417
[08:03:29]  <GameMaker> Johnanneswilm: Talking a bout this issue, there've been some updates
[08:03:41]  jsbell (~jsbell@a11e3e9c.public.cloak) joined the channel
[08:03:41]  <GameMaker> Sanket: Could we use the exsting API?
[08:03:59]  <GameMaker> sanketj The thing that delays that is that there is only one place to pass a promist
[08:04:08]  <GameMaker> sanketj so there is not way to actually delay the execustion
[08:04:16]  <GameMaker> sanketj based on that we need to make an upstae
[08:04:26]  alexk (~alexk@a11e3e9c.public.cloak) joined the channel
[08:04:29]  <GameMaker> sanketj One idea is a map of callbacks
[08:04:43]  <GameMaker> sanketj a smaller change is to pass in an option
[08:04:54]  <GameMaker> sanketj the 2nd option is it's better egonomics
[08:05:04]  estade (~estade@a11e3e9c.public.cloak) joined the channel
[08:05:07]  <GameMaker> sanketj and it make is so we don't have to have the same thing specified in two places
[08:05:11]  <GameMaker> sanketj other thoughs?
[08:05:40]  <GameMaker> johanneswilm Option 1 is linked to an explainer, there is an option 2 on that page, is that the 2nd option?
[08:05:49]  <GameMaker> sanketj no, that's a variation on option 1
[08:06:13]  <jsbell> apologies for interruption - I got a late invite - is there a voice call as well as IRC? If so can someone point me to connection details?
[08:06:34]  <GameMaker> whsieh yes, option 2 seems the better options
[08:06:48]  <GameMaker> jsbell yes - https://meet.jit.si/W3CEditingCall
[08:07:19]  <GameMaker> sanketj it needs to be a promise to a blob, or a promise to a funciton
[08:07:45]  <GameMaker> sanketj in this case, we'd pass a promise to a function, and it would hold onto it
[08:08:05]  <GameMaker> johanneswilm from a web author's perspective, why do we have a promise that is immediatly resolved
[08:08:30]  <GameMaker> whsieh that is how it works, there's no opportunitu to make it lazy
[08:08:43]  <GameMaker> sanketj ergonomically it's still convientent 
[08:09:23]  <GameMaker> sanketj any other thoughts? Josh, Evan others?
[08:10:11]  <GameMaker> estade maybe I'm not understanding, but there doens't seem to be a change to the api
[08:10:41]  <GameMaker> jsbell there was an update that the promise would be egarly resolved, so a callback is needed
[08:10:52]  <GameMaker> estade it doesn't need to be a callback that returns a promise?
[08:11:04]  <GameMaker> sanketj we want to delay the execution of that function 
[08:11:24]  <GameMaker> estade it's good to have more options, and its good to nudge authors in the right direciton
[08:11:47]  <GameMaker> estade even if we introduce it, there's nothting that compels an author to use it
[08:11:56]  <GameMaker> estade I see two big issues
[08:12:15]  <GameMaker> estade we'd hve to do something when we have to forward as written the data, copy, close tab, then paste
[08:12:25]  <GameMaker> estade it expose to the user that something is going on under the hood
[08:12:39]  <GameMaker> estade if we go with some with a dialog, then that is. more useable
[08:13:04]  <GameMaker> estade the second thing, if you immediately start an async task, but then it's not expensvie to the user
[08:13:18]  <GameMaker> estade by the time they paste, then it's done by the time the user pastes
[08:13:47]  <GameMaker> estade if you delay at paste, then the user doens't think that it is responsive, and will paste again
[08:14:15]  <GameMaker> sanketj what is expensive - the partner team has a seneriou where they write several formats tot he clipboard
[08:14:41]  <GameMaker> sanketj some are custom formats, to generate the payload, they reach out to a server, populate the data, and the it comes back
[08:15:14]  <GameMaker> sanketj on paste, some users just need one or two of those formates, and they do all that work for no reason
[08:15:27]  <GameMaker> sanketj if it's delayed, then they only need to do that call when it's asked form
[08:16:11]  <GameMaker> snianu if excell online generates a custom format, only excell understands that format
[08:16:35]  <GameMaker> snianu web custom formates are very expensive to produce, it's very heavy, very expensive
[08:16:52]  <GameMaker> snianu even if it's async, it's still expensive and may not be even needed
[08:17:08]  <GameMaker> estade what is expsnvies? cpu, bandwith, etc?
[08:17:31]  <GameMaker> snianu it's all of those, the cost if very expensive for the webauthors, in the case for Excell 9
[08:18:06]  <GameMaker> sanketj for CPU, you will pay that cost on paste, but because it's so expsnsive it seems worth to delay
[08:18:46]  <GameMaker> jsbell We need to make sure that we say what we mean by 'expensive'
[08:19:07]  <GameMaker> jsbell we should be clear why we delay, why this is better for the user
[08:19:19]  <GameMaker> sanketj sorry for not updating the explainer
[08:19:26]  <GameMaker> sanketj we'll update the background there
[08:19:52]  <GameMaker> sanketj if we look at the native implementation, they do use delayed clipboard renederend because some of the formats are expensive
[08:20:30]  <GameMaker> sanketj if we close the tab, what we notice from the apps that we've tried, then the app will only keep around the cheap formats, and not keep around the expensive formats
[08:20:55]  <GameMaker> sanketj the apps that we're using have their own document model, they have their own wya to know if there was a change in the range that was copied
[08:21:13]  <GameMaker> sanketj the current API will allow them to make changes, or make a 2nd write
[08:21:28]  <GameMaker> estade do aps have the ability to write whenever they want to? 
[08:21:52]  <GameMaker> sanketj if the app notices that it's being closed, they can generate some basic formats and write them in a non-deferred manner
[08:22:05]  <GameMaker> estade they can only do that if there's a user action
[08:22:32]  <GameMaker> sanketj yes, only if there's a user actions, but the apps have their own way of determining if there's been a change in the area that has been copied
[08:23:01]  <GameMaker> estade I think that when we're making decsion about web apis, we need to be first concerned about the user
[08:23:36]  <GameMaker> estade and if we're encougaing apps to do something to not have the experience that the user expects, that seems wrong
[08:24:06]  <GameMaker> sanketj yes, but many native apps have delayed rendering, there are situations where certian formats are not going to be there, and they won't be available
[08:25:25]  <GameMaker> snianu technically in delayed rendereing, web authors don't write to the full format, when the callback is executed, if there's a callback associated with it, and the browswer retieves that call back, during that, the web authors produce the payload, then we set that data for the formate, there's not user gesture restriction, if you have the callback, you get the payload
[08:25:36]  <GameMaker> snianu but it doesn't answer your content change questions
[08:25:54]  <GameMaker> snianu the webauthor's don't have to write this
[08:26:18]  <GameMaker> johanneswilm is there's cliboard history, can you go back three items on the shipping OSes? 
[08:27:13]  <GameMaker> snianu the system cliboard keeps track of all the formats, so if you copy three formats, and the 2nd copy there's four formats, the first is verision 0, and the second is version 1, the OS knows how to keep track for everything
[08:27:26]  <GameMaker> snianu the OS keeps track of all that
[08:27:42]  <GameMaker> johanneswilm if we go back three items, we don't call the functions
[08:28:09]  <GameMaker> sanketj the window's cliboard history feature, they only save the text format
[08:28:22]  <GameMaker> sanketj if you do that, it will do it at render
[08:29:01]  <GameMaker> jsbell speaking of text, do we want to special case text as a format, the UA could specify the text foram
[08:29:14]  <GameMaker> jsbell require that at least one of the formats not be delay rendered?
[08:29:34]  <GameMaker> jsbell if on shutdown if everything is delayed rendered
[08:30:20]  <GameMaker> estade I can think of one situation when you would want to delay renerding for text, chrom is not very good at selecting megabytes of text, it's slow
[08:30:42]  <GameMaker> estade even when we add this functionaltiy, we should direct to websauthoers that this isn't the best way to optomise
[08:31:14]  <GameMaker> estade say there's 5 formats, if you could just download once and converge
[08:31:26]  <GameMaker> estade if the user never pastes, that doesn't seem a big deal to me
[08:31:55]  <GameMaker> estade what's the ration of copiest to paste? probably not 10 to 1, probably more like2 to 1
[08:32:06]  <GameMaker> estade if there's 5 things that need to be generated on the suerver
[08:32:27]  <jsbell> snianu [via chat] Just a small correction: Currently, the built-in clipboard history only supports text, HTML, and images less than 4 MB in size.
[08:32:30]  <GameMaker> estade it would be good to go back to see why native apps use this 
[08:33:27]  <GameMaker> estade I rememebr when we were first building chrome, we did not find this to be a good tradeoff, it could be deferred, at the time, we determined it wouldn't be a good tradeoff
[08:33:50]  <GameMaker> estade native apps don't have the problem, it wouldn't be a big issues to use local CPU cycles
[08:34:04]  <GameMaker> estade in short, is it not possibel for the computation to be pushed to the client? 
[08:34:11]  <GameMaker> estade is that not a possibility? 
[08:34:24]  <GameMaker> estade are we really painted into a corner in order to need to implemnent this API
[08:34:43]  <GameMaker> estade I think it will be a worse user experience, even if it's necessary
[08:34:59]  <GameMaker> sanketj even if you do all the optimiattions on the server, it can still be expensvie
[08:35:09]  <GameMaker> sanketj if you're a very popualr web app, 
[08:35:28]  <GameMaker> sanketj I dn't think we are necessariy trying to encourge or discourage delayed renedering
[08:35:43]  <GameMaker> sanketj from the user, the not delayed experinece, is bettter
[08:36:01]  <GameMaker> sanketj but delayed clipboard rendering does help the apps sometimes
[08:36:14]  <GameMaker> sanketj we aren't forcing the use of delayed rendering
[08:36:23]  <GameMaker> sanketj but this allowes the app to make that tradeoff
[08:36:36]  <GameMaker> sanketj we're also only modlifying the async clipboard api
[08:36:46]  <GameMaker> sanketj not the other one that would have more impact
[08:37:03]  <GameMaker> estade the platform should be somewhat oppinionated
[08:37:25]  <GameMaker> estade we don't want to say this is optomised when there are tradeoffs
[08:37:40]  <GameMaker> estade what's the status, are they slow, or are they holding features back?
[08:37:59]  <GameMaker> sanketj features are being held back becaue it's just too expenive without delayed clipboard rendering
[08:38:19]  <GameMaker> sanketj the expenives formats are also not used, so it seems like an acceptable tradeoff
[08:38:50]  <GameMaker> snianu on excell online, if it takes too long, the paste fails, it shows a message that the app is unable to render the data
[08:39:01]  <GameMaker> johanneswilm is that happening server side?
[08:39:19]  <GameMaker> snianu yes if they server isn't able to make the payload quickly enough
[08:39:34]  <GameMaker> snianu and if this with the normal API
[08:39:55]  <GameMaker> snianu at least with promises, we wait for all the promises to resolve
[08:40:12]  <GameMaker> snianu if we're not able to product the data is x seconds, then it fails, 
[08:40:24]  <GameMaker> snianu it's not worth all the back and forth and they timout the paste
[08:40:51]  <GameMaker> estade I know there's optimizaiton in chromium to make the async clipboard api
[08:41:14]  <GameMaker> estade a webapp should work more and more like a native app, it should still be able to work like a native app
[08:41:24]  <GameMaker> estade you should still be able to copy and paste quickly
[08:41:44]  <GameMaker> estade is this only relegvant to web apps that have given up on offline experineces
[08:42:34]  <GameMaker> sanketj no, it's the other way around, even some of these native apps, have become more and more of a connected experience, syncing your documents from one place to another, there's a good amount of data that's processed ont he server rather than on the client even in native apps
[08:42:50]  <GameMaker> sanketj not everything can be done int he client, in both a native and a webapp
[08:43:17]  <GameMaker> estade I find that disheartening, it would be nice to be able to use the app even if I'm on an airplane
[08:43:43]  <GameMaker> estade maybe we'll solve internet connectivity, but for now, all things should be moveing towards working better with a shakey or not internet connection
[08:44:03]  <GameMaker> estade what's the copy paste esperience when you're teathered to your cell phone and you ahve a bad connection?
[08:44:28]  <GameMaker> estade if every copy opperations requires a heavy sever componenet
[08:44:38]  <GameMaker> estade as an app authoer I'd be trying to offload more of that to the client
[08:44:58]  <GameMaker> sanketj basic functionality will not stop working, but it's more about the complext formats
[08:45:10]  <GameMaker> sanketj more and more of those things will be powered by a web experience.
[08:46:29]  <GameMaker> sanketj to summerise our point, I think we, I think the current API surface is to not forcet he web in one way or ther other. if there's more way to be explicit in how we delay render. there's a deisre to make things work more like native
[08:46:58]  <GameMaker> johanneswilm make it would make sense of having some real life examples of where this is needed. having real examples that cannot be transformed in client, where you need these server calls. 
[08:47:26]  <GameMaker> johanneswilm if we have such examples, then maybe we can move forward, other wise it's just back and forth on whether this is needed
[08:47:38]  <GameMaker> sanketj we can find out more information on where this is needed
[08:48:02]  <GameMaker> sanketj the native apps do use delayed clipboard rendering, and there is no way to do this on the web today
[08:48:21]  <GameMaker> estade more concrete examples help make the case
[08:48:40]  <GameMaker> estade a way to make the case even stronger, something that preserver some of the delay
[08:48:57]  <GameMaker> estade your custom format can be jsut enough data for the other app to fetch the data itself
[08:49:09]  <GameMaker> estade this means there isn't a probelm with closing the app
[08:49:16]  <GameMaker> estade this is the one though
[08:49:28]  <GameMaker> estade adding that, or other alternatives can really help
[08:49:43]  <GameMaker> estade besieds the arguemtn of this is how it's been done in the paste
[08:49:50]  <GameMaker> estade you don't want to reinvent the wheel
[08:50:16]  <GameMaker> johanneswilm sanketj can you find some examples
[08:50:30]  <GameMaker> sanketj estade could you provice more exmaples of a better experinece?
[08:50:46]  <GameMaker> estade the directive we follow is to put the end user first
[08:50:52]  <GameMaker> estade then the webapp auther
[08:50:57]  <GameMaker> estade then user agent laast
[08:51:18]  <GameMaker> estade we've identified in some ways that a async but eager copy can make for a better exerinece
[08:51:41]  <GameMaker> estade if you can give some sort of token where the receiivne app can get the date themselves
[08:52:12]  <GameMaker> estade it's potentially a better end user experience that can be explored
[08:52:21]  <GameMaker> estade that's what I'm focusing on, is the user
[08:52:39]  <GameMaker> estade and of course it's not a good user experinece for them to pay for downloading data when they paste
[08:52:57]  <GameMaker> johanneswilm anyone else?
[08:53:20]  <GameMaker> sanketj we are limited a bit based on what the OS cliboard offers, if there's not a way to put tokens, then we can't do that.
[08:53:40]  <GameMaker> sanketj we can think of some other solution, but we need to integrate with the OS clipboard.
[08:54:13]  <GameMaker> johanneswilm ok, until next time, sanketj try to find examples for this, and we recovien in a month.
[08:54:43]  <GameMaker> alexk we have a pr in review for and have a comple more questions
[08:55:04]  <GameMaker> alexk good news for the project, google has more recources they can give us
[08:55:21]  <GameMaker> alexk #33 in the edit contect repo
[08:55:22] -Github- https://github.com/w3c/editing/issues/33 : Should execCommand be spec'd to do nothing in cE=typing?
[08:55:47]  <GameMaker> johanneswilm this is still continuing but slow
[08:55:48]  <GameMaker> ?
[08:56:06]  <GameMaker> alexk that's it for edit context
[08:56:17]  estade (~estade@a11e3e9c.public.cloak) left IRC (Ping timeout: 180 seconds)
[08:56:38]  <GameMaker> sanketj I just wanted to say that given that the next working group call is a month away, is there any way we can resolve and make some progress in the issue
[08:57:13]  <GameMaker> johanneswilm some history, we just did things on an e-mail list, we can do that in the meantime
[08:57:33]  <GameMaker> johanneswilm there's nothing that says we have to do things in these calls, the e-mail list can make things more faster
[08:57:47]  <GameMaker> johanneswilm put your things on github, and then send and e-mail saying you want feedback
[08:58:09]  <GameMaker> sanketj sounds great, even, josh, and wenson, anyone from firefox? 
[08:58:14]  <GameMaker> sanketj Ali right? 
[08:58:22]  <johanneswilm> https://lists.w3.org/Archives/Public/public-editing-tf/

-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3c/editing/issues/417#issuecomment-1462421392
You are receiving this because you are subscribed to this thread.

Message ID: <w3c/editing/issues/417/1462421392@github.com>