[Houdini] Minutes Toronto F2F 2019-06-07 Part I: Worklets

=================================================
   These are the official Houdini Task Force
     minutes. Unless you're correcting the
      minutes, please respond by starting
 a new thread with an appropriate subject line.
=================================================


Worklets
--------

  - RESOLVED: Handle binary data by making a <blob> type in P&V, so a
              custom property can be set to a blob, and string-om can
              serialize it as `blob(...blob-url...)`; this utilizes
              the Transferable mechanism to handle snapshotting/
              invalidating. (Or possibly a slightly different KV
              store, will research this.) (Issue #872)
  - RESOLVED: Add input properties, expose for the target, should work
              for input right now. And we'll pursue events / message
              channel later for output. (Issue #869)

===== FULL MINUTES BELOW ======

Agenda: https://github.com/w3c/css-houdini-drafts/wiki/Toronto-F2F-June-2019

Present:
  Rossen Atanassov, Microsoft
  Tab Atkins, Google
  L. David Baron, Mozilla
  Amelia Bellamy-Royds, Invited Expert
  Rick Byers, Google
  Tantek Çelik, Mozilla
  Rob Flack, Google
  Jihye Hong, LGE
  Brian Kardell, Igalia and Open JS Foundation
  Ian Kilpatrick, Google
  Stanton Marcum, Amazon
  Myles C. Maxfield, Apple
  Cameron McCormack, Mozilla
  Giorgio Natili, Amazon
  Theresa O'Connor, Apple
  François Remy, Invited Expert
  Alan Stearns, Adobe
  Majid Valipour, Google

Scribe: iank

Worklets
========

Passing binary data to worklets, implications on P&V and TypedOM
----------------------------------------------------------------
  github: https://github.com/w3c/css-houdini-drafts/issues/872

  TabAtkins: So several of the different houdini spec, folks when they
             use them, have use cases for passing large amounts of
             binary data to worklets. E.g. vertex data to a paint
             worklet, etc.
  TabAtkins: Its enough data that stringifying and passing through a
             custom prop is a bit of a performance issue, and we
             expect this to get worse over time.
  TabAtkins: We need a way to send typed arrays into worklets.
  TabAtkins: So we'll go through a few of the options.

  TabAtkins: First off - creating a new properties and values type
             which takes binary data, in the typedOM this would take
             an binary array, in the string APIs there probably isn't
             a reasonable serialization.
  TabAtkins: So that possibly works, there is a question around what
             happens with mutations.
  TabAtkins: E.g. do we need to snapshot it, then compare?
  TabAtkins: Feels straightforward from a design perspective, but lots
             of corner cases.
  TabAtkins: 2nd possibility, we hook up post message. This requires
             some new APIs, so they can be informed, or possibly it
             hooks up to a map, and we call you again when a new
             message has been issued. This sounds promising, because
             the contract looks simpler, postMessage has already
             transferrables, works on SABs etc.
  TabAtkins: Seems pretty promising, but we need to work on an API
             surface that would work.
  TabAtkins: Was there any other options?
  <dbaron> I wonder how that interacts with worklet lifetime

  majidvp: You could do it on a per API basis, e.g. animation worklet
           has an options bag, which could be updated to allow you to
           pass in more data later.
  flackr: With paint worklet
  flackr: ... I'm not sure this will work as there can be multiple
          instances.
  flackr: Easier for the first option, however this first option
          doesn't work with transferrables.

  Rossen: Are there any implementors with experience?
  majidvp: Audio worklet uses postmessage to do this.
  Rossen: What's the feedback so far.
  majidvp: This satisfies the use cases.
  iank: This works for audio worklet as they have one instance and
        they have a very strict processing model.
  <astearns> anyone have a link for the audio worklet solution?
  heycam: Not sure its a good idea, is it possible for blobs and blob
          uris?
  TabAtkins: Today the current model is base64.
  iank: TypedOM version of the base-uri?
  heycam: That's not what I was saying, but I think I prefer if we can
          get the postmessage solution to work.
  heycam: postmessage, what you really want is you call a function on
          the worklet to set a key-value into a map, and with
          postmessage its difficult to respond to it as occurs outside
          the other worklet functions.
  heycam: You could replay the messages as other instances spin up?

  dbaron: So one of my concerns is that - you can destroy and
          re-create these worklets, and this doesn't seem compatible
          with the postmessage APIs.
  dbaron: You could do something where all the messages go into a
          store which persists
  rbyers: This sounds comparable to the typedOM.
  dbaron: When you shift between worklets you need to copy between the
          instances.

  myles: I just wanted to bring up the fact the use case the webgpu
         sends binary data to another process. They might be going to
         do this with checkpointing. Would be good if these solutions
         were similar.

  TabAtkins: Does anyone have experience with AudioWorklet, or should
             I go talk to the editors?
  majidvp: There is an instance and send message to that instance.

  flackr: On the first idea, around having something in the typed OM,
          if you had a const underlying array, you could pass a
          reference to this around, and not have the copy overhead.
  TabAtkins: That would be a new typed array API.
  flackr: A const typed array....

  emilio: The issue with TypedOM is that you need to define how to
          serialize, e.g. calling the string APIs.
  TabAtkins: It might be ok that the string APIs just produce
             something like "[blob]"
  TabAtkins: Referring to global named defining constructs inherited
             into the shadow dom.
  emilio: You could, instead of stashing the thing into typedom
          value - produce the name, have an extra function which is
          "blob(...)"
  TabAtkins: At what point are you describing a different API to the
             TypedOM get property API?
  TabAtkins: If you put blobs in a k-v store, ... how many k-v stores
             do you want to expose?
  emilio: This won't put the blob on the TypedOM?
  TabAtkins: Why not?
  emilio: But then you need to define how to serialize...
  emilio: Its basically the same issue as the props within the shadow
          apis.
  TabAtkins: I don't like breaking the invariant that you can access
             the value and re-set this?
  heycam: I feel like that's not a lot of work.
  TabAtkins: We could serialize to a base64 value....
  emilio: The issue is that you are creating many copies,... but not
          going to die on this hill.
  flackr: Could serialize to a blob-url....
  TabAtkins: A blob-uri is a short representation to the array, which
             might work.
  TabAtkins: Register a blob-uri on the document global...
  emilio: Registered custom properties are global for the whole
          document...
  TabAtkins: You have to be careful with name collisions, however
             blob-uris don't have that issue.

  flackr: One thing that you have to be careful, is that you need to
          know if they are valid or not, so that we can destroy them
          when necessary.
  TabAtkins: hmm... yeah lifetime issues are interesting.

  rbyers: Maybe this has been covered, the fundamental tension is that
          worklets are designed to be stateless... and people want to
          message state.
  rbyers: Seems architecture is preferable that this state flows
          through the TypedOM...
  majidvp: One thing with ports is that you can pass them around. E.g.
           audio worklet plumbing into paint worklet to drive it.
  majidvp: But I don't know how strong of a use case this is...
  TabAtkins: As long as you have a proper invalidation channel.
  iank: But there can be many paint worklets.
  TabAtkins: We might want to look at transient binary data, but that
             doesn't stay around.
  flackr: The audio one use case falls down, as could trigger repaint
          due to resize.
  heycam: With blobs you need to keep them around until you use the
          string...
  TabAtkins: I was talking before about shared-array-buffer... you
             still need the way for the audio worklet to drive the
             invalidation.
  flackr: But we need the SAB in the typedOM right?
  TabAtkins: yes...
  TabAtkins: That seems like the best way to do winamp visualization.
  rbyers: How do you do synchronization?
  majidvp: Could use atomics?
  flackr: I think its up to the author to build the synchronization.

  <talking about painting while a SAB is changing data while
      underneath>
  TabAtkins: For the main use case the an ArrayBuffer value type seems
             like the right way and do with custom properties?
  rbyers: This has to be const... for the array buffer.
  <people seem to agree>
  TabAtkins: It might be worthwhile to do a separate K-V store, e.g. I
             depend on these keys vs. I depend on these properties.

  flackr: blob-uris have the contract with the typed array, but does
          require some one work to track references to them.
  rbyers: In most of these use cases i'm not sure you want to keep
          around older data.
  flackr: Thats true for winamp, however for lottie renderer you set
          the data once.
  TabAtkins: blob-uris are for blobs within your process, these are
             potentially not.
  flackr: Conceptually these are an immutable shared array buffer for
          these cases.
  bkardell: This seems better for offscreen canvas.
  TabAtkins: Can audioworklet postmessage back?
  iank: Yes.
  TabAtkins: Lets just talk about static binary data then.
  majidvp: A port is bi-directional and can pass them around.

  heycam: Would you ever want to animate these custom properties which
          are these arrays?
  TabAtkins: Everything can be animated, and then you do a binary flip.
  TabAtkins: If it's a registered prop, you set a transition using
             this property... oh you couldn't do a transition/
             animation because it doesn't have a string apis,... oh
             but you can if we use blob-uris.
  heycam: Was just thinking about from the perspective from the
          animation if this works.
  iank: The problem is the key is a string

  heycam: What is the lifecycle for the blob-uris?
  <discussion around revokedatauri and how this works..>
  flackr: TypedOM would always have a reference to the data even if
          revoke data uri was called.

  heycam: Are the string versions going to be the url() function?
  TabAtkins: No there will be blob() function as don't want to
             conflate with network access.
  heycam: On the other end, on the worklet api how do you read the
          data from the typed array?
  TabAtkins: The typedOM would expose the typed array directly.
  heycam: Oh I thought you'd have an extra step to get the blob-uri,
          then get the data.
  TabAtkins: Seems like an extra step for no good reason?
  flackr: If the typed om took a url, but the value was only set after
          the url was fetched, you could plug in arbitary urls.
  bkardell: There seems like there are pieces within the platform
  TabAtkins: Has issues with things from the fetch api, e.g. can't
             fetch w/ credentials, etc.
  TabAtkins: Would prefer to keep this simple.
  bkardell: There seems like there are pieces within the platform
  TabAtkins: We should fix url however...
  flackr: Tab are you saying that we could upgrade to a full url
          eventually?
  TabAtkins: You could create a typedOM way to create a credentialed
             request etc.
  TabAtkins: We plan on exposing custom hooks for functions to
             transform specified values into computed at some point.

  TabAtkins: <summary of discussion> Future planned api should allow
             us to handle urls in custom props. without too much extra
             complication, and not changing our current discussion.
  Rossen: It sounds like we are done with this issue.
  TabAtkins: Lets get a resolution for this is how we are going to
             design it.

  majidvp: Animation worklet is slightly different, as there is an
           options bag. But I think this proposal works nicely.
  majidvp: Only thing that is missing is invalidating the options.
  iank: This discussion is just about plumbing array data into the
        typed om and effects this has
  iank: We can consider animationworklet separately like audioworklet
        did
  majidvp: If we consider options to be parallel to the stylemap to
           paint worklet... if you hook up an invalidation method,
           this would work. and we can add message passing.
  majidvp: Just wanted to point out the parallel, and it could work
           there as well.

  bkardell: I support the idea of being consistent as possible
  <TabAtkins> proposed resolution: Handle binary data by making a
              <blob> type in P&V, so a custom property can be set to a
              blob, and string-om can serialize it as
              `blob(...blob-url...)`; this utilizes the Transferable
              mechanism to handle snapshotting/invalidating.
  heycam: Still have a suspicion that an explicit K-V thing would be
          easier to work...
  flackr: I really don't thing we need a K-V store however, the key is
          the property, the value is the typed object.
  heycam: You are right that the custom props are a K-V, but they have
          additional requirements.
  <TabAtkins> proposed resolution: Handle binary data by making a
              <blob> type in P&V, so a custom property can be set to a
              blob, and string-om can serialize it as
              `blob(...blob-url...)`; this utilizes the Transferable
              mechanism to handle snapshotting/invalidating. (Or
              possibly a slightly different KV store, will research
              this.)

  rbyers: Is this something we could try out with some customers?
  <folks - yes>
  Rossen: Any objections?

  RESOLVED: Handle binary data by making a <blob> type in P&V, so a
            custom property can be set to a blob, and string-om can
            serialize it as `blob(...blob-url...)`; this utilizes the
            Transferable mechanism to handle snapshotting/
            invalidating. (Or possibly a slightly different KV store,
            will research this.)

  Rossen: Does that cover all three of these issues?
  TabAtkins: The post message issue was about paint worklet. and
             agreed that right now use offscreen canvas, and if we
             need it later can consider.

Sending data to animators in worklet from main thread
-----------------------------------------------------
  github: https://github.com/w3c/css-houdini-drafts/issues/869

  majidvp: Similar issue as previous issue, we want to get some data
           from the main thread to the animation thread. We have this
           at startup with the options bag.
  majidvp: The issue is that there might be new data on the main
           thread that affect the animation, e.g. sizes of elements
           change.
  majidvp: One idea is that do allow the options to be mutable, the
           main thread can up date the options, and it'll update the
           options on the animation thread.
  majidvp: The other options is using a postmessage port with the
           animator.

  flackr: If we want harmonize with css - we could do this with a
          property map.
  TabAtkins: Yes.
  flackr: There are several elements in play, and could read
          properties out of those elements.
  TabAtkins: When the animate() function get called, are there element
             references?
  majidvp: No the effect has a target, we don't expose that right now,
           but we could expose a styleMap.
  TabAtkins: You'd want to limit the number of properties.
  flackr: You could just expose all the same properties?
  hober: I agree.

  TabAtkins: Seems like we just want to expose inputProperties on the
             target for the effect? It doesn't let you pass
             information out however.
  majidvp: There are use cases for more stateful effects, and this
           stage I'm fine with not solving this.
  majidvp: These use cases are more important once you feed input
           events into animation worklet. But we don't have this yet,
           so its not pressing.

  bkardell: A message channel is good for JS, but not good for CSS?
  iank: another option is event dispatching
  majidvp: Output could be events which fits nicely with the
           animations API.
  majidvp: postmessage is more generic, but has its own lifetime
           concerns, but you might not want to commit to that.
  TabAtkins: Proposal: Add input properties, expose for the target,
             should work for input right now. And we'll pursue events/
             message channel later for output.

  RESOLVED: Add input properties, expose for the target, should work
            for input right now. And we'll pursue events / message
            channel later for output.

  [break 15 mins]

Received on Thursday, 11 July 2019 23:12:57 UTC