- From: CSS Meeting Bot via GitHub <sysbot+gh@w3.org>
- Date: Fri, 07 Jun 2019 14:20:55 +0000
- To: public-houdini-archive@w3.org
The Houdini Task Force just discussed `cheaply passing binary data to worklets, and the implications on P&V and TypedOM`, and agreed to the following: * ``RESOLVED: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)`` <details><summary>The full IRC log of that discussion</summary> <iank_> Topic: cheaply passing binary data to worklets, and the implications on P&V and TypedOM<br> <TabAtkins> github: https://github.com/w3c/css-houdini-drafts/issues/872<br> <iank_> TabAtkins: So several of the different houdini spec, folks when they use them, have usecases for passing large amounts of binary data to worklets. E.g. vertex data to a paint worklet, etc.<br> <iank_> TabAtkins: Its enough data that stringifying and passing through a custom prop is a bit of a performance issues, and we expect this to get worse over time.<br> <iank_> TabAtkins: We need a way to send typed arrays into worklets.<br> <iank_> TabAtkins: So we'll go through a few of the options.<br> <iank_> TabAtkins: First off - creating a new properties and values type which takes binary data, in the typedOM this would take an binary array, in the string APIs there probably isn't a reasonable serialization.<br> <giorgionatili> Giorgio Natili, Amazon<br> <iank_> TabAtkins: So that possibly works, there is a question around what happens with mutations.<br> <iank_> TabAtkins: E.g. do we need to snapshot it, then compare?<br> <iank_> TabAtkins: Feels straightforward from a design perspective, but lots of corner cases.<br> <iank_> TabAtkins: 2nd possibility, we hook up post message. This requires some new APIs, so they can be informed, or possibly it hooks up to a map, and we call you again when a new message has been issued. This sounds promising, b/c the contract looks simpler, postMessage has already transferrables, works on SABs etc.<br> <iank_> TabAtkins: Seems pretty promising, but we need to work on an API surface that would work.<br> <iank_> TabAtkins: Was there any other options?<br> <dbaron> I wonder how that interacts with worklet lifetime<br> <iank_> majidvp: You could do it on a per API basis, e.g. animation worklet has an options bag, which could be used.<br> <heycam> q+<br> <iank_> flackr: With paint worklet i<br> <iank_> flackr: ... i'm not sure this will work as there can be multiple instances.<br> <TabAtkins> s/be used/be updated to allow you to pass in more data later/<br> <iank_> flackr: Easier for the first option, however this first option doesn't work with transferrables.<br> <iank_> Rossen_: Are there any impl experience?<br> <iank_> majidvp: Audio worklet uses postmessage to do this.<br> <myles_> q+<br> <iank_> Rossen_: Whats the feedback so far.<br> <iank_> majidvp: This satifies the usecases.<br> <iank_> iank_: This works for audio worklet as they have one instance and they have a very strict processing model.<br> <astearns> anyone have a link for the audio worklet solution?<br> <rbyers> q?<br> <flackr> q+<br> <iank_> heycam: Not sure its a good idea, is it possible for blobs and blob uris?<br> <iank_> TabAtkins: Today the current model is base64 it.<br> <Rossen_> ack heycam<br> <iank_> iank_: TypedOM version of the base-uri?<br> <iank_> heycam: That's not what i was saying, but I think I perfer if we can get the postmessage solution to work.<br> <iank_> heycam: postmessage, what you really want is you call a function on the worklet to set a key-value into a map, and with postmessage its difficult to respond to it as occurs outside the other worklet functions.<br> <iank_> heycam: You could replay the messages as other instances spin up?<br> <iank_> dbaron: So one of my concerns is that - you can destory and re-create these worklets, and this doesn't seem compatible with the postmessage APIs.<br> <iank_> dbaron: You could do something where all the messages go into a store which persists<br> <iank_> rbyers: This sounds comparable to the typedOM.<br> <emilio> q+<br> <iank_> dbaron: When you shift between worklets you need to copy between the instances.<br> <rbyers> q+<br> <Rossen_> ack myles_<br> <iank_> myles_: I just wanted to bring up the fact the usecase the webgpu sends binary data to another process. They are going to do this with checkpointing. Would be good if these solutions were similar.<br> <iank_> TabAtkins: Does anyone have experience?<br> <myles_> s/are/might/<br> <TabAtkins> s/experience/experience with AudioWorklet, or should I go talk to the editors/<br> <iank_> majidvp: There is an instance and send message to that instance.<br> <Rossen_> ack flackr<br> <myles_> TabAtkins: Rafael Cintron at Microsoft<br> <iank_> flackr: On the first idea, around having something in the typed OM, if you had a const underlying array, you could pass a reference to this around, and not have the copy overhead.<br> <iank_> TabAtkins: That would be a new TypedOM API.<br> <iank_> flackr: A const typed array....<br> <TabAtkins> s/TypedOM/typed array/<br> <iank_> emilio: The issue with TypedOM is that you need to define how to serialize, e.g. calling the string APIs.<br> <iank_> TabAtkins: It might be ok that the string APIs just produce something like "[blob]"<br> <iank_> TabAtkins: referring to global named defining constructs inherited into the shadow dom.<br> <iank_> emilio: you could, instead of stashing the thing into typedom value - produce the name, have an extra function which is "blob(...)"<br> <iank_> TabAtkins: At what point are you describing a different API to the TypedOM get property API?<br> <iank_> TabAtkins: If you put blobs in a k-v store, ... how many k-v stores do you want to expose?<br> <iank_> emilio: This won't but the blob on the TypedOM?<br> <iank_> TabAtkins: Why not?<br> <iank_> emilio: But then you need to define how to serialize...<br> <iank_> emilio: Its basically the same issue as the props within the shadow apis.<br> <iank_> TabAtkins: I don't like breaking the invariant that you can access the value and re-set this?<br> <iank_> heycam: I feel like thats not a lot of work.<br> <iank_> TabAtkins: we could serialize to a base64 value....<br> <iank_> emilio: the issue is that you are creating many copies,... but not going to die on this hill.<br> <iank_> flackr: Could serialize to a blob-url....<br> <iank_> TabAtkins: A blob-uri is a short represenation to the array, which might work.<br> <iank_> TabAtkins: Register a blob-uri on the document global...<br> <iank_> emilio: blob-uris are global for the whole document.....<br> <iank_> TabAtkins: You have to be careful with name collisions, however blob-uris don't have that issue.<br> <iank_> flackr: One thing that you have to be careful, is that you need to know if they are valid or not, so that we can destory them when nessecery.<br> <emilio> s/blob-uris/registered custom properties/ on my comment above<br> <iank_> TabAtkins: hmm... yeah lifetime issues are interesting.<br> <Rossen_> q?<br> <Rossen_> ack emilio<br> <iank_> rbyers: Maybe this has been covered, the fundemental tension is that worklets are designed to be stateless... and people want to message state.<br> <majidvp> q+<br> <iank_> rbyers: Seems arch. preferable that this state flows through the TypedOM...<br> <iank_> rbyers: Anyone disagree?<br> <iank_> majidvp: One thing with ports is that you can pass them around. E.g. audio worklet plumbing into paint worklet to drive it.<br> <iank_> majidvp: But I don't know how strong of a usecase this is...<br> <iank_> TabAtkins: As long as you have a proper invalidation channel.<br> <iank_> iank_: But there can be many paint worklets.<br> <iank_> TabAtkins: We might want to look at transient binary data, but that doesn't stay around.<br> <iank_> flackr: The audio one usecase falls down, as could trigger repaint due to resize.<br> <iank_> heycam: With blobs you need to keep them around until you use the string....<br> <iank_> TabAtkins: I was talking before about shared-array-buffer... you still need the way for the audio worklet to drive the invalidation.<br> <iank_> flackr: But we need the SAB in the typedOM right?<br> <iank_> TabAtkins: yes...<br> <iank_> TabAtkins: That seems like the best way to do winamp viz.<br> <iank_> rbyers: How do you do synchronization?<br> <iank_> majidvp: Could use atomics?<br> <iank_> flackr: I think its up to the author to build the synchronization.<br> <bkardell_> q+<br> <iank_> (talking about painting while a SAB is chaning data while underneath).<br> <iank_> TabAtkins: For the main usecase the an ArrayBuffer value type seems like the right way and do with custom properties?<br> <iank_> rbyers: This has to be const... for the array buffer.<br> <iank_> <people seem to agree><br> <iank_> TabAtkins: It might be worthwhile to do a separate K-V store, e.g. I depend on these keys vs. I depend on these properties.<br> <iank_> flackr: blob-uris have the contract with the typed array, but does require some one work to track references to them.<br> <Rossen_> q?<br> <Rossen_> ack rbyers<br> <iank_> rbyers: In most of these usecases i'm not sure you want to keep around older data.<br> <iank_> flackr: Thats true for winamp, however for lottie renderer you set the data once.<br> <iank_> TabAtkins: blob-uris are for blobs within your process, these are potentially not.<br> <iank_> flackr: Conceptually these are an immutable shared array buffer for these cases.<br> <iank_> bkardell_: This seems better for offscreen canvas.<br> <iank_> TabAtkins: Can audioworklet postmessage back?<br> <iank_> iank_: Yes.<br> <iank_> TabAtkins: Lets just talk about static binary data then.<br> <Rossen_> ack majidvp<br> <heycam> q+<br> <bkardell_> q-<br> <iank_> majidvp: A port is di-directional and can pass them around.<br> <Rossen_> ack heycam<br> <iank_> heycam: Would you ever want to animate these custom properties which are these arrays?<br> <iank_> TabAtkins: Everything can be animated, and then you do a binary flip.<br> <iank_> TabAtkins: If its a registered prop, you set a transition using this property... oh you couldn't do a transition/animation b/c it doesn't have a string apis,... oh but you can if we use blob-uris.<br> <iank_> heycam: Was just thinking about from the perspective from the animation if this works.<br> <flackr> iank_: the problem is the key is a string<br> <iank_> heycam: What is the lifecycle for the blob-uris?<br> <Rossen_> q?<br> <iank_> <discussion around revokedatauri and how this works..><br> <iank_> flackr: TypedOM would always have a reference to the data even if revoke data uri was called.<br> <iank_> heycam: Are the string versions going to be the url() function?<br> <iank_> TabAtkins: no there will be blob() function as don't want to conflate with network access .<br> <iank_> heycam: On the other end, on the worklet api how do you read the data from the typed array?<br> <iank_> TabAtkins: The typedOM would expose the typed array directly.<br> <iank_> heycam: Oh i thought you'd have an extra step to get the blob-uri, then get the data.<br> <iank_> TabAtkins: Seems like an extra step for no good reason?<br> <iank_> bkardell_: blob blob blob blob<br> <Rossen_> s/bkardell_: blob blob blob blob//<br> <iank_> flackr: If the typed om took a url, but the value was only set after the url was fetched, you could plug in arbitary urls.<br> <iank_> bkardell_: There seems like there are peices within the platofrm<br> <iank_> TabAtkins: Has issues with things from the fetch api, e.g. can't fetch w/ credentials, etc.<br> <iank_> TabAtkins: Would prefer to keep this simple.<br> <iank_> bkardell_: There seems like there are peices within the platofrm<br> <iank_> TabAtkins: We should fix url however...<br> <iank_> flackr: Tab are you saying that we could upgrade to a full url eventually?<br> <iank_> TabAtkins: You could create a typedom way to create a credentialed request etc.<br> <iank_> TabAtkins: We plan on exposing custom hooks for fucntions to transform specified values into compuated at some point.<br> <iank_> TabAtkins: <summary of discussion> Future planned api should allow us to handle urls in custom props. without too much extra complication, and not changing our current discussion.<br> <majidvp> q+<br> <iank_> Rossen_: It sounds like we are done with this issue.<br> <iank_> TabAtkins: Lets get a resolution for this is how we are going to design it.<br> <iank_> majidvp: animation worklet is slightly different, as there is an options bag. but i think this proposal works nicely.<br> <iank_> majidvp: only thing that is missing is invalidating the options.<br> <flackr> iank_: this discussion is just about plumbing array data into the typed om and effects this has<br> <flackr> iank_: we can consider animationworklet separately like audioworklet did<br> <bkardell_> q+<br> <iank_> majidvp: If we consider options to be parallel to the stylemap to paint worklet... if you hook up an invalidation method, this would work. and we can add message passing.<br> <iank_> majidvp: Just wanted to point out the parallel, and it could work there as well.<br> <Rossen_> ack *<br> <iank_> bkardell_: I support the idea of being consistent as possible<br> <Rossen_> ack majidvp<br> <TabAtkins> proposed resolution: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating.<br> <bkardell_> q-<br> <iank_> bkardell_: Its a weak statement however.<br> <Rossen_> ack bkardell_<br> <iank_> heycam: Still have a suspicsion that a explicit K-V thing would be easier to work...<br> <iank_> flackr: I really don't thing we need a K-V store however, the key is the property, the value is the typed object.<br> <iank_> heycam: you are right that the custom props are a K-V, but they have additional requirements.<br> <TabAtkins> proposed resolution: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)<br> <iank_> rbyers: Is this something we could try out with some customers?<br> <iank_> <folks - yes><br> <iank_> Rossen_: Any objections?<br> <iank_> Rossen_: Resolved.<br> <iank_> Rossen_: Does that cover all three of these issues?<br> <iank_> Resolved: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)<br> <iank_> RESOLVED: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)<br> <iank_> TabAtkins: The post message issue was about paint worklet. and agreed that right now use offscreen canvas, and if we need it later can consider.<br> </details> -- GitHub Notification of comment by css-meeting-bot Please view or discuss this issue at https://github.com/w3c/css-houdini-drafts/issues/872#issuecomment-499903766 using your GitHub account
Received on Friday, 7 June 2019 14:20:58 UTC