Re: Using the Web Audio API in Web Workers

These notes would be good to have in the Worker issue on the spec.  Could
you paste them into https://github.com/WebAudio/web-audio-api/issues/16?

On Wed, Apr 8, 2015 at 1:54 PM, Joe Berkovitz <joe@noteflight.com> wrote:

> Hi Ashley,
>
> What I can tell you is that it's definitely come up a lot, and was even
> discussed on the most recent conference call since we were already talking
> about the need for WebMIDI access in workers.
>
> Your implementation ideas are also helpful. Right now we are weighing the
> priorities of the things that will make the biggest immediate difference in
> the utility of the spec. Having this input makes it easier to understand
> how audio in Web Workers might be done, and who it would affect.
>
> Best
> ...Joe
>
> On Wed, Apr 8, 2015 at 12:11 PM, Ashley Gullen <ashley@scirra.com> wrote:
>
>> We develop a HTML5 game engine and our goal is to ultimately have the
>> entire engine running from a Web Worker, off the main thread.
>>
>> There is progress on getting canvas rendering from workers specced. The
>> last major hurdle to running an entire game engine in a worker is audio
>> playback. Currently AFAIK Web Audio is only specced to work from the main
>> thread, and it is a relatively complex API, so if a game engine is running
>> in a worker there would need to be a bridge between the worker and the main
>> thread with lots of postMessage going on, which could impact performance
>> and add latency while adding complexity. Having Web Audio available in Web
>> Workers would solve this. Has this been investigated yet?
>>
>> Ideally AudioContext could be made available in WorkerGlobalScope.
>> However I can immediately see a number of issues:
>>
>> - integration with Media Elements would not be available (e.g.
>> createMediaElementSource), because they are part of the DOM which is not
>> available in a Worker
>> - integration with Media Streams would not be available, including from
>> getUserMedia, since these are not available in a Worker (AFAIK)
>> - selection of audio input and output devices is currently only available
>> on the main thread (although this may not be a problem if device ID strings
>> can be posted to the worker)
>> - ideally types like AudioBuffer would be Transferrable
>>
>> Implementation ideas:
>> - a WorkerAudioContext with the unsupported features removed (e.g.
>> anything using media elements or media streams) but otherwise working
>> identically to AudioContext
>> - a WorkerAudioContext with all features, but some way of integrating
>> with media elements/media streams from the main thread (e.g. by ID
>> strings/object URLs)
>> - a WorkerAudioContext with some kind of specialised WorkerMediaStream
>> that can access an object URL created from the main thread and route audio
>> from either a media element or media stream to the WorkerAudioContext
>> - a WorkerAudioContext with all features, including speccing Media
>> Streams for workers! (possibly very difficult)
>> - a Transferrable AudioContextProxy that can be obtained from an existing
>> AudioContext on the main thread and sent to a worker. Calls to
>> AudioContextProxy would simply be forwarded to the existing AudioContext
>> similar to what a postMessage implementation would do, so it's really using
>> the same context. AudioContextProxy would also be missing media
>> element/media stream features, but these could still be implemented from
>> the main thread since it can use all features there. This is probably the
>> simplest solution and significantly reduces the burden of bridging the APIs
>> in JS (reducing the bridge to only the features necessary to cover media
>> elements/media streams), but in the long term implementing MediaStreams in
>> workers seems like it would be desirable too.
>>
>> Any thoughts?
>>
>> Ashley Gullen
>> Scirra.com
>>
>>
>
>
> --
> .            .       .    .  . ...Joe
>
> *Joe Berkovitz*
> President
>
> *Noteflight LLC*
> Boston, Mass.
> phone: +1 978 314 6271
> www.noteflight.com
> "Your music, everywhere"
>