Re: Reviewing the Web Audio API (from webrtc)

Hi Chris,

thanks for quick feedback. I would just want to point out that my 
feedback was my personal, not for the webrtc WG. I hope we can come with 
WG feedback later on.

Stefan

On 03/29/2012 04:33 AM, Chris Rogers wrote:
> Hi Stefan, thanks for you comments.  I'll try to give some answers
> inline below.
>
> For reference, please refer to this provisional document describing
> scenarios for WebRTC / WebAudio interaction.
> This is the same one that I posted to the list earlier last year and
> presented to the WebRTC working group at TPAC 2011:
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html
>
> As we discussed at TPAC, this document would need some small changes to
> be able to handle MediaStreamTrack objects explicitly instead of
> MediaStreams.  Please note that these extensions to the Web Audio API
> (from the webrtc-integration  document above) are not *yet* in the main
> Web Audio API specification document.  I would like some help from the
> WebRTC group to help finish this part of the API (from
> the webrtc-integration.html document above).
>
> Please see rest of comments inline below.
>
> Thanks,
> Chris
>
> On Wed, Mar 28, 2012 at 7:05 PM, Wei, James <james.wei@intel.com
> <mailto:james.wei@intel.com>> wrote:
>
>     From: Stefan Hakansson LK <stefan.lk.hakansson@ericsson.com
>     <mailto:stefan.lk.hakansson@ericsson.com>> ____
>
>     Date: Wed, 28 Mar 2012 20:54:38 +0200____
>
>     Message-ID: <4F735E6E.7070706@ericsson.com
>     <mailto:4F735E6E.7070706@ericsson.com>> ____
>
>     To: "public-webrtc@w3.org <mailto:public-webrtc@w3.org>"
>     <public-webrtc@w3.org <mailto:public-webrtc@w3.org>> ____
>
>     I've spent some time looking at the Web Audio API [1] since the
>     Audio WG ____
>
>     have put out a call for review [2].____
>
>     __ __
>
>     As starting point I used the related reqs from our use-case and req ____
>
>     document [3]:____
>
>     __ __
>
>     F13             The browser MUST be able to apply spatialization____
>
>                          effects to audio streams.____
>
>     __ __
>
>     F14             The browser MUST be able to measure the level____
>
>                          in audio streams.____
>
>     F15             The browser MUST be able to change the level____
>
>                          in audio streams.____
>
>     __ __
>
>     with the accompanying API reqs:____
>
>     __ __
>
>     A13             The Web API MUST provide means for the web____
>
>                          application to apply spatialization effects to____
>
>                          audio streams.____
>
>     A14             The Web API MUST provide means for the web____
>
>                          application to detect the level in audio____
>
>                          streams.____
>
>     A15             The Web API MUST provide means for the web____
>
>                          application to adjust the level in audio____
>
>                          streams.____
>
>     A16             The Web API MUST provide means for the web____
>
>                          application to mix audio streams.____
>
>     __ __
>
>     Looking at the Web Audio API, and combining it with the MediaStream ____
>
>     concept we use, I come to the following understanding:____
>
>     __ __
>
>     1) To make the audio track(s) of MediaStream(s) available to the Web
>     ____
>
>     Audio processing blocks, the The MediaElementAudioSourceNode
>     Interface ____
>
>     would be used.
>
>
> Generally, it would be a "MediaStreamAudioSourceNode" (or perhaps
> "MediaStreamTrackAudioSourceNode"), please see example 3:
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html
>
>     ____
>
>     __ __
>
>     2) Once that is done, the audio is available to the Web Audio API ____
>
>     toolbox, and anything we have requirements on can be done____
>
>     __ __
>
>     3) When the processing has been done (panning, measure level, change
>     ____
>
>     level, mix) the audio would be played using an AudioDestinationNode ____
>
>     Interface
>
>
> The AudioDestinationNode represents the local client's "speakers", in
> example 3 a MediaStream is created with the line:
> var peer = context.createMediaStreamDestination();
>
> to send the processed audio to the remote peer.
>
>     ____
>
>     __ __
>
>     What is unclear to me at present, is how synchronization would work.
>     So ____
>
>     far we have been discussing in terms of that all tracks in a
>     MediaStream ____
>
>     are kept in sync; but what happens when the audio tracks are routed
>     to ____
>
>     another set of tools, and not played in the same (video) element as
>     the ____
>
>     video?
>
>
> HTMLMediaElements already have a mechanism for synchronization using the
> HTML5 MediaController API.  Live stream (live camera/mic or remote
> peers) MediaStreams would maintain synchronization (I assume you mean
> audio/video sync in this case).  The Web Audio API would just be used to
> apply effects, not changing the synchronization.
>
>     ____
>
>     __ __
>
>     Another take away is that the processing can only happen in the
>     browser ____
>
>     that is going to play the audio, since there is no way to go from an
>     ____
>
>     AudioNode to a MediaStream or MediaStreamTrack.
>
>
> There are some examples showing how to go from an AudioNode to
> MediaStream (needs to be developed for MediaStreamTrack).
> Please look especially at the use of the createMediaStreamSource()
> and createMediaStreamDestination() methods.
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html
>
>     ____
>
>     __ __
>
>     Anyone else that has looked into the Web Audio API? And any other ____
>
>     conclusions?____
>
>     __ __
>
>     I think we should give feedback from this WG (as we have some reqs
>     that ____
>
>     are relevant).____
>
>     __ __
>
>     Br,____
>
>     Stefan____
>
>     __ __
>
>     __ __
>
>     [1] http://www.w3.org/TR/2012/WD-webaudio-20120315/ ____
>
>     [2]
>     http://lists.w3.org/Archives/Public/public-webrtc/2012Mar/0072.html ____
>
>     [3] ____
>
>     http://datatracker.ietf.org/doc/draft-ietf-rtcweb-use-cases-and-requirements/?include_text=1
>     ____
>
>     __ __
>
>     Best Regards ____
>
>     __ __
>
>     James ____
>
>     __ __
>
>     __ __
>
>

Received on Thursday, 29 March 2012 06:15:20 UTC