- From: Dael Jackson <daelcss@gmail.com>
- Date: Thu, 11 Jul 2019 19:11:54 -0400
- To: public-houdini@w3.org
- Cc: www-style@w3.org
=================================================
These are the official Houdini Task Force
minutes. Unless you're correcting the
minutes, please respond by starting
a new thread with an appropriate subject line.
=================================================
Worklets
--------
- RESOLVED: Handle binary data by making a <blob> type in P&V, so a
custom property can be set to a blob, and string-om can
serialize it as `blob(...blob-url...)`; this utilizes
the Transferable mechanism to handle snapshotting/
invalidating. (Or possibly a slightly different KV
store, will research this.) (Issue #872)
- RESOLVED: Add input properties, expose for the target, should work
for input right now. And we'll pursue events / message
channel later for output. (Issue #869)
===== FULL MINUTES BELOW ======
Agenda: https://github.com/w3c/css-houdini-drafts/wiki/Toronto-F2F-June-2019
Present:
Rossen Atanassov, Microsoft
Tab Atkins, Google
L. David Baron, Mozilla
Amelia Bellamy-Royds, Invited Expert
Rick Byers, Google
Tantek Çelik, Mozilla
Rob Flack, Google
Jihye Hong, LGE
Brian Kardell, Igalia and Open JS Foundation
Ian Kilpatrick, Google
Stanton Marcum, Amazon
Myles C. Maxfield, Apple
Cameron McCormack, Mozilla
Giorgio Natili, Amazon
Theresa O'Connor, Apple
François Remy, Invited Expert
Alan Stearns, Adobe
Majid Valipour, Google
Scribe: iank
Worklets
========
Passing binary data to worklets, implications on P&V and TypedOM
----------------------------------------------------------------
github: https://github.com/w3c/css-houdini-drafts/issues/872
TabAtkins: So several of the different houdini spec, folks when they
use them, have use cases for passing large amounts of
binary data to worklets. E.g. vertex data to a paint
worklet, etc.
TabAtkins: Its enough data that stringifying and passing through a
custom prop is a bit of a performance issue, and we
expect this to get worse over time.
TabAtkins: We need a way to send typed arrays into worklets.
TabAtkins: So we'll go through a few of the options.
TabAtkins: First off - creating a new properties and values type
which takes binary data, in the typedOM this would take
an binary array, in the string APIs there probably isn't
a reasonable serialization.
TabAtkins: So that possibly works, there is a question around what
happens with mutations.
TabAtkins: E.g. do we need to snapshot it, then compare?
TabAtkins: Feels straightforward from a design perspective, but lots
of corner cases.
TabAtkins: 2nd possibility, we hook up post message. This requires
some new APIs, so they can be informed, or possibly it
hooks up to a map, and we call you again when a new
message has been issued. This sounds promising, because
the contract looks simpler, postMessage has already
transferrables, works on SABs etc.
TabAtkins: Seems pretty promising, but we need to work on an API
surface that would work.
TabAtkins: Was there any other options?
<dbaron> I wonder how that interacts with worklet lifetime
majidvp: You could do it on a per API basis, e.g. animation worklet
has an options bag, which could be updated to allow you to
pass in more data later.
flackr: With paint worklet
flackr: ... I'm not sure this will work as there can be multiple
instances.
flackr: Easier for the first option, however this first option
doesn't work with transferrables.
Rossen: Are there any implementors with experience?
majidvp: Audio worklet uses postmessage to do this.
Rossen: What's the feedback so far.
majidvp: This satisfies the use cases.
iank: This works for audio worklet as they have one instance and
they have a very strict processing model.
<astearns> anyone have a link for the audio worklet solution?
heycam: Not sure its a good idea, is it possible for blobs and blob
uris?
TabAtkins: Today the current model is base64.
iank: TypedOM version of the base-uri?
heycam: That's not what I was saying, but I think I prefer if we can
get the postmessage solution to work.
heycam: postmessage, what you really want is you call a function on
the worklet to set a key-value into a map, and with
postmessage its difficult to respond to it as occurs outside
the other worklet functions.
heycam: You could replay the messages as other instances spin up?
dbaron: So one of my concerns is that - you can destroy and
re-create these worklets, and this doesn't seem compatible
with the postmessage APIs.
dbaron: You could do something where all the messages go into a
store which persists
rbyers: This sounds comparable to the typedOM.
dbaron: When you shift between worklets you need to copy between the
instances.
myles: I just wanted to bring up the fact the use case the webgpu
sends binary data to another process. They might be going to
do this with checkpointing. Would be good if these solutions
were similar.
TabAtkins: Does anyone have experience with AudioWorklet, or should
I go talk to the editors?
majidvp: There is an instance and send message to that instance.
flackr: On the first idea, around having something in the typed OM,
if you had a const underlying array, you could pass a
reference to this around, and not have the copy overhead.
TabAtkins: That would be a new typed array API.
flackr: A const typed array....
emilio: The issue with TypedOM is that you need to define how to
serialize, e.g. calling the string APIs.
TabAtkins: It might be ok that the string APIs just produce
something like "[blob]"
TabAtkins: Referring to global named defining constructs inherited
into the shadow dom.
emilio: You could, instead of stashing the thing into typedom
value - produce the name, have an extra function which is
"blob(...)"
TabAtkins: At what point are you describing a different API to the
TypedOM get property API?
TabAtkins: If you put blobs in a k-v store, ... how many k-v stores
do you want to expose?
emilio: This won't put the blob on the TypedOM?
TabAtkins: Why not?
emilio: But then you need to define how to serialize...
emilio: Its basically the same issue as the props within the shadow
apis.
TabAtkins: I don't like breaking the invariant that you can access
the value and re-set this?
heycam: I feel like that's not a lot of work.
TabAtkins: We could serialize to a base64 value....
emilio: The issue is that you are creating many copies,... but not
going to die on this hill.
flackr: Could serialize to a blob-url....
TabAtkins: A blob-uri is a short representation to the array, which
might work.
TabAtkins: Register a blob-uri on the document global...
emilio: Registered custom properties are global for the whole
document...
TabAtkins: You have to be careful with name collisions, however
blob-uris don't have that issue.
flackr: One thing that you have to be careful, is that you need to
know if they are valid or not, so that we can destroy them
when necessary.
TabAtkins: hmm... yeah lifetime issues are interesting.
rbyers: Maybe this has been covered, the fundamental tension is that
worklets are designed to be stateless... and people want to
message state.
rbyers: Seems architecture is preferable that this state flows
through the TypedOM...
majidvp: One thing with ports is that you can pass them around. E.g.
audio worklet plumbing into paint worklet to drive it.
majidvp: But I don't know how strong of a use case this is...
TabAtkins: As long as you have a proper invalidation channel.
iank: But there can be many paint worklets.
TabAtkins: We might want to look at transient binary data, but that
doesn't stay around.
flackr: The audio one use case falls down, as could trigger repaint
due to resize.
heycam: With blobs you need to keep them around until you use the
string...
TabAtkins: I was talking before about shared-array-buffer... you
still need the way for the audio worklet to drive the
invalidation.
flackr: But we need the SAB in the typedOM right?
TabAtkins: yes...
TabAtkins: That seems like the best way to do winamp visualization.
rbyers: How do you do synchronization?
majidvp: Could use atomics?
flackr: I think its up to the author to build the synchronization.
<talking about painting while a SAB is changing data while
underneath>
TabAtkins: For the main use case the an ArrayBuffer value type seems
like the right way and do with custom properties?
rbyers: This has to be const... for the array buffer.
<people seem to agree>
TabAtkins: It might be worthwhile to do a separate K-V store, e.g. I
depend on these keys vs. I depend on these properties.
flackr: blob-uris have the contract with the typed array, but does
require some one work to track references to them.
rbyers: In most of these use cases i'm not sure you want to keep
around older data.
flackr: Thats true for winamp, however for lottie renderer you set
the data once.
TabAtkins: blob-uris are for blobs within your process, these are
potentially not.
flackr: Conceptually these are an immutable shared array buffer for
these cases.
bkardell: This seems better for offscreen canvas.
TabAtkins: Can audioworklet postmessage back?
iank: Yes.
TabAtkins: Lets just talk about static binary data then.
majidvp: A port is bi-directional and can pass them around.
heycam: Would you ever want to animate these custom properties which
are these arrays?
TabAtkins: Everything can be animated, and then you do a binary flip.
TabAtkins: If it's a registered prop, you set a transition using
this property... oh you couldn't do a transition/
animation because it doesn't have a string apis,... oh
but you can if we use blob-uris.
heycam: Was just thinking about from the perspective from the
animation if this works.
iank: The problem is the key is a string
heycam: What is the lifecycle for the blob-uris?
<discussion around revokedatauri and how this works..>
flackr: TypedOM would always have a reference to the data even if
revoke data uri was called.
heycam: Are the string versions going to be the url() function?
TabAtkins: No there will be blob() function as don't want to
conflate with network access.
heycam: On the other end, on the worklet api how do you read the
data from the typed array?
TabAtkins: The typedOM would expose the typed array directly.
heycam: Oh I thought you'd have an extra step to get the blob-uri,
then get the data.
TabAtkins: Seems like an extra step for no good reason?
flackr: If the typed om took a url, but the value was only set after
the url was fetched, you could plug in arbitary urls.
bkardell: There seems like there are pieces within the platform
TabAtkins: Has issues with things from the fetch api, e.g. can't
fetch w/ credentials, etc.
TabAtkins: Would prefer to keep this simple.
bkardell: There seems like there are pieces within the platform
TabAtkins: We should fix url however...
flackr: Tab are you saying that we could upgrade to a full url
eventually?
TabAtkins: You could create a typedOM way to create a credentialed
request etc.
TabAtkins: We plan on exposing custom hooks for functions to
transform specified values into computed at some point.
TabAtkins: <summary of discussion> Future planned api should allow
us to handle urls in custom props. without too much extra
complication, and not changing our current discussion.
Rossen: It sounds like we are done with this issue.
TabAtkins: Lets get a resolution for this is how we are going to
design it.
majidvp: Animation worklet is slightly different, as there is an
options bag. But I think this proposal works nicely.
majidvp: Only thing that is missing is invalidating the options.
iank: This discussion is just about plumbing array data into the
typed om and effects this has
iank: We can consider animationworklet separately like audioworklet
did
majidvp: If we consider options to be parallel to the stylemap to
paint worklet... if you hook up an invalidation method,
this would work. and we can add message passing.
majidvp: Just wanted to point out the parallel, and it could work
there as well.
bkardell: I support the idea of being consistent as possible
<TabAtkins> proposed resolution: Handle binary data by making a
<blob> type in P&V, so a custom property can be set to a
blob, and string-om can serialize it as
`blob(...blob-url...)`; this utilizes the Transferable
mechanism to handle snapshotting/invalidating.
heycam: Still have a suspicion that an explicit K-V thing would be
easier to work...
flackr: I really don't thing we need a K-V store however, the key is
the property, the value is the typed object.
heycam: You are right that the custom props are a K-V, but they have
additional requirements.
<TabAtkins> proposed resolution: Handle binary data by making a
<blob> type in P&V, so a custom property can be set to a
blob, and string-om can serialize it as
`blob(...blob-url...)`; this utilizes the Transferable
mechanism to handle snapshotting/invalidating. (Or
possibly a slightly different KV store, will research
this.)
rbyers: Is this something we could try out with some customers?
<folks - yes>
Rossen: Any objections?
RESOLVED: Handle binary data by making a <blob> type in P&V, so a
custom property can be set to a blob, and string-om can
serialize it as `blob(...blob-url...)`; this utilizes the
Transferable mechanism to handle snapshotting/
invalidating. (Or possibly a slightly different KV store,
will research this.)
Rossen: Does that cover all three of these issues?
TabAtkins: The post message issue was about paint worklet. and
agreed that right now use offscreen canvas, and if we
need it later can consider.
Sending data to animators in worklet from main thread
-----------------------------------------------------
github: https://github.com/w3c/css-houdini-drafts/issues/869
majidvp: Similar issue as previous issue, we want to get some data
from the main thread to the animation thread. We have this
at startup with the options bag.
majidvp: The issue is that there might be new data on the main
thread that affect the animation, e.g. sizes of elements
change.
majidvp: One idea is that do allow the options to be mutable, the
main thread can up date the options, and it'll update the
options on the animation thread.
majidvp: The other options is using a postmessage port with the
animator.
flackr: If we want harmonize with css - we could do this with a
property map.
TabAtkins: Yes.
flackr: There are several elements in play, and could read
properties out of those elements.
TabAtkins: When the animate() function get called, are there element
references?
majidvp: No the effect has a target, we don't expose that right now,
but we could expose a styleMap.
TabAtkins: You'd want to limit the number of properties.
flackr: You could just expose all the same properties?
hober: I agree.
TabAtkins: Seems like we just want to expose inputProperties on the
target for the effect? It doesn't let you pass
information out however.
majidvp: There are use cases for more stateful effects, and this
stage I'm fine with not solving this.
majidvp: These use cases are more important once you feed input
events into animation worklet. But we don't have this yet,
so its not pressing.
bkardell: A message channel is good for JS, but not good for CSS?
iank: another option is event dispatching
majidvp: Output could be events which fits nicely with the
animations API.
majidvp: postmessage is more generic, but has its own lifetime
concerns, but you might not want to commit to that.
TabAtkins: Proposal: Add input properties, expose for the target,
should work for input right now. And we'll pursue events/
message channel later for output.
RESOLVED: Add input properties, expose for the target, should work
for input right now. And we'll pursue events / message
channel later for output.
[break 15 mins]
Received on Thursday, 11 July 2019 23:12:56 UTC