Re: [mediacapture-transform] Out-of-main-thread processing by default (#23)

> > Audio worklet is one example.

I think this is a perfect example, and a strong corollary to video:
* [ScriptProcessorNode](https://webaudio.github.io/web-audio-api/#ScriptProcessorNode) exposed audio data on the main-thread
* Everyone agreed this was a terrible mistake, and deprecated it, slating it for removal.
* Interesting trivia: Its immediate replacement was [initially going to be](http://www.devdoc.net/web/developer.mozilla.org/en-US/docs/Web/API/ScriptProcessorNode.html) "Audio Workers"
* But the Audio WG found this improvement insufficient, further isolating exposure to a highly controlled [AudioWorklet]() environment.

There's clear and relevant precedent here for controlling exposure of realtime data away from main thread, possibly even away from workers.

> I was referring specifically to APIs exposed on DedicatedWorker, but not on Window.

@guidou Is the argument we shouldn't expose methods to workers without also exposing them to main-thread, because that would be unexpected somehow? What's the worry there exactly? If symmetry is of concern, we can do VideoWorklet. 😉

> How would a worklet support the use cases that require generating a new track in JS?
In particular, the use case where you want to generate a new track based on the contents of two (or more) existing tracks (i.e., the weather report use case).

It can be done with audio today, so I don't think whether it is possible or not is the discussion.

Instead, worker vs. worklet I think comes down to what benefits there may be from controlling the environment of exposure.

I don't know what those are atm, but if we plan to expose `VideoFrame`s from a GPU buffer pool https://github.com/w3c/webcodecs/issues/83 we might wish we had some control, so JS failing to close them quickly doesn't stall camera capture or WebGL in the browser.

-- 
GitHub Notification of comment by jan-ivar
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/23#issuecomment-842585817 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 17 May 2021 19:48:02 UTC