Re: Using the Web Audio API in Web Workers

Alex - I'd say the question is there all right and is still very
significant as an architectural issue, but it's not truly forced by Worker
support. I think that what is being asked for here is a Worker-based
AudioContext whose output will be mixed into the (default) audio
destination, much like any other AudioContext in the main thread.

On Wed, Apr 8, 2015 at 7:17 PM, Chris Wilson <cwilso@google.com> wrote:

> Yes.  Although realistically, the default output (and input) of Web Audio
> isn't going in/out from the main thread anyway - it's going in and out of
> the audio thread, and if one believes <audio> data is in the main thread we
> already have this problem.  :)
>
> On Wed, Apr 8, 2015 at 3:47 PM, Alex Russell <slightlyoff@google.com>
> wrote:
>
>> It sounds like this would force the question about "what is the default
>> output of Web Audio anyway?". As long as it has only been available from
>> the main thread, this hasn't been pressing, but now if we make this
>> available in workers (and, perhaps as a result, available inside
>> sub-workers constructed from Service Workers), we need to understand how to
>> select/attach to an output.
>>
>> On Wed, Apr 8, 2015 at 2:14 PM, Joe Berkovitz <joe@noteflight.com> wrote:
>>
>>> Thanks, Chris -- Done.
>>>
>>> On Wed, Apr 8, 2015 at 4:57 PM, Chris Wilson <cwilso@google.com> wrote:
>>>
>>>> These notes would be good to have in the Worker issue on the spec.
>>>> Could you paste them into
>>>> https://github.com/WebAudio/web-audio-api/issues/16?
>>>>
>>>> On Wed, Apr 8, 2015 at 1:54 PM, Joe Berkovitz <joe@noteflight.com>
>>>> wrote:
>>>>
>>>>> Hi Ashley,
>>>>>
>>>>> What I can tell you is that it's definitely come up a lot, and was
>>>>> even discussed on the most recent conference call since we were already
>>>>> talking about the need for WebMIDI access in workers.
>>>>>
>>>>> Your implementation ideas are also helpful. Right now we are weighing
>>>>> the priorities of the things that will make the biggest immediate
>>>>> difference in the utility of the spec. Having this input makes it easier to
>>>>> understand how audio in Web Workers might be done, and who it would affect.
>>>>>
>>>>> Best
>>>>> ...Joe
>>>>>
>>>>> On Wed, Apr 8, 2015 at 12:11 PM, Ashley Gullen <ashley@scirra.com>
>>>>> wrote:
>>>>>
>>>>>> We develop a HTML5 game engine and our goal is to ultimately have the
>>>>>> entire engine running from a Web Worker, off the main thread.
>>>>>>
>>>>>> There is progress on getting canvas rendering from workers specced.
>>>>>> The last major hurdle to running an entire game engine in a worker is audio
>>>>>> playback. Currently AFAIK Web Audio is only specced to work from the main
>>>>>> thread, and it is a relatively complex API, so if a game engine is running
>>>>>> in a worker there would need to be a bridge between the worker and the main
>>>>>> thread with lots of postMessage going on, which could impact performance
>>>>>> and add latency while adding complexity. Having Web Audio available in Web
>>>>>> Workers would solve this. Has this been investigated yet?
>>>>>>
>>>>>> Ideally AudioContext could be made available in WorkerGlobalScope.
>>>>>> However I can immediately see a number of issues:
>>>>>>
>>>>>> - integration with Media Elements would not be available (e.g.
>>>>>> createMediaElementSource), because they are part of the DOM which is not
>>>>>> available in a Worker
>>>>>> - integration with Media Streams would not be available, including
>>>>>> from getUserMedia, since these are not available in a Worker (AFAIK)
>>>>>> - selection of audio input and output devices is currently only
>>>>>> available on the main thread (although this may not be a problem if device
>>>>>> ID strings can be posted to the worker)
>>>>>> - ideally types like AudioBuffer would be Transferrable
>>>>>>
>>>>>> Implementation ideas:
>>>>>> - a WorkerAudioContext with the unsupported features removed (e.g.
>>>>>> anything using media elements or media streams) but otherwise working
>>>>>> identically to AudioContext
>>>>>> - a WorkerAudioContext with all features, but some way of integrating
>>>>>> with media elements/media streams from the main thread (e.g. by ID
>>>>>> strings/object URLs)
>>>>>> - a WorkerAudioContext with some kind of specialised
>>>>>> WorkerMediaStream that can access an object URL created from the main
>>>>>> thread and route audio from either a media element or media stream to the
>>>>>> WorkerAudioContext
>>>>>> - a WorkerAudioContext with all features, including speccing Media
>>>>>> Streams for workers! (possibly very difficult)
>>>>>> - a Transferrable AudioContextProxy that can be obtained from an
>>>>>> existing AudioContext on the main thread and sent to a worker. Calls to
>>>>>> AudioContextProxy would simply be forwarded to the existing AudioContext
>>>>>> similar to what a postMessage implementation would do, so it's really using
>>>>>> the same context. AudioContextProxy would also be missing media
>>>>>> element/media stream features, but these could still be implemented from
>>>>>> the main thread since it can use all features there. This is probably the
>>>>>> simplest solution and significantly reduces the burden of bridging the APIs
>>>>>> in JS (reducing the bridge to only the features necessary to cover media
>>>>>> elements/media streams), but in the long term implementing MediaStreams in
>>>>>> workers seems like it would be desirable too.
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>> Ashley Gullen
>>>>>> Scirra.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> .            .       .    .  . ...Joe
>>>>>
>>>>> *Joe Berkovitz*
>>>>> President
>>>>>
>>>>> *Noteflight LLC*
>>>>> Boston, Mass.
>>>>> phone: +1 978 314 6271
>>>>> www.noteflight.com
>>>>> "Your music, everywhere"
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> .            .       .    .  . ...Joe
>>>
>>> *Joe Berkovitz*
>>> President
>>>
>>> *Noteflight LLC*
>>> Boston, Mass.
>>> phone: +1 978 314 6271
>>> www.noteflight.com
>>> "Your music, everywhere"
>>>
>>
>>
>


-- 
.            .       .    .  . ...Joe

*Joe Berkovitz*
President

*Noteflight LLC*
Boston, Mass.
phone: +1 978 314 6271
www.noteflight.com
"Your music, everywhere"