Re: Reviewing the Web Audio API (from webrtc)

Hi Chris,

now I have had the time to read your mail as well!

I have a couple of follow up questions:

* for the time being, when no createMediaStreamSource or Destination 
methods are available, can the outline I used be used (i.e. would it 
work)? (1. Use MediaElementAudioSourceNode to get access to audio; 2. 
process using Web Audio API methods; 3. Play using AudioDestinationNode)

* perhaps createMediaStreamSource / Destination should work on track 
level instead (as you seem to indicate as well); a MediaStream is really 
just a collection of tracks, and those can be audio or video tracks. If 
you work on track level you can do processing that results in an audio 
track and combine that with a video track into a MediaStream

* kind of schedule do you foresee for the things needed to integrate 
with MediaStreams (tracks)?

Thanks,
Stefan

PS There seem to be two proposals in the Audio WG; one is the Web Audio 
API, the other MediaStream Processing API. Probably needless to say, I 
want to stay out of any selection/comparison discussion. This is only 
feedback/comments on the Web Audio API.

On 03/29/2012 04:33 AM, Chris Rogers wrote:
> Hi Stefan, thanks for you comments.  I'll try to give some answers
> inline below.
>
> For reference, please refer to this provisional document describing
> scenarios for WebRTC / WebAudio interaction.
> This is the same one that I posted to the list earlier last year and
> presented to the WebRTC working group at TPAC 2011:
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html
>
> As we discussed at TPAC, this document would need some small changes to
> be able to handle MediaStreamTrack objects explicitly instead of
> MediaStreams.  Please note that these extensions to the Web Audio API
> (from the webrtc-integration  document above) are not *yet* in the main
> Web Audio API specification document.  I would like some help from the
> WebRTC group to help finish this part of the API (from
> the webrtc-integration.html document above).
>
> Please see rest of comments inline below.
>
> Thanks,
> Chris
>
> On Wed, Mar 28, 2012 at 7:05 PM, Wei, James <james.wei@intel.com
> <mailto:james.wei@intel.com>> wrote:
>
>     From: Stefan Hakansson LK <stefan.lk.hakansson@ericsson.com
>     <mailto:stefan.lk.hakansson@ericsson.com>> ____
>
>     Date: Wed, 28 Mar 2012 20:54:38 +0200____
>
>     Message-ID: <4F735E6E.7070706@ericsson.com
>     <mailto:4F735E6E.7070706@ericsson.com>> ____
>
>     To: "public-webrtc@w3.org <mailto:public-webrtc@w3.org>"
>     <public-webrtc@w3.org <mailto:public-webrtc@w3.org>> ____
>
>     I've spent some time looking at the Web Audio API [1] since the
>     Audio WG ____
>
>     have put out a call for review [2].____
>
>     __ __
>
>     As starting point I used the related reqs from our use-case and req ____
>
>     document [3]:____
>
>     __ __
>
>     F13             The browser MUST be able to apply spatialization____
>
>                          effects to audio streams.____
>
>     __ __
>
>     F14             The browser MUST be able to measure the level____
>
>                          in audio streams.____
>
>     F15             The browser MUST be able to change the level____
>
>                          in audio streams.____
>
>     __ __
>
>     with the accompanying API reqs:____
>
>     __ __
>
>     A13             The Web API MUST provide means for the web____
>
>                          application to apply spatialization effects to____
>
>                          audio streams.____
>
>     A14             The Web API MUST provide means for the web____
>
>                          application to detect the level in audio____
>
>                          streams.____
>
>     A15             The Web API MUST provide means for the web____
>
>                          application to adjust the level in audio____
>
>                          streams.____
>
>     A16             The Web API MUST provide means for the web____
>
>                          application to mix audio streams.____
>
>     __ __
>
>     Looking at the Web Audio API, and combining it with the MediaStream ____
>
>     concept we use, I come to the following understanding:____
>
>     __ __
>
>     1) To make the audio track(s) of MediaStream(s) available to the Web
>     ____
>
>     Audio processing blocks, the The MediaElementAudioSourceNode
>     Interface ____
>
>     would be used.
>
>
> Generally, it would be a "MediaStreamAudioSourceNode" (or perhaps
> "MediaStreamTrackAudioSourceNode"), please see example 3:
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html
>
>     ____
>
>     __ __
>
>     2) Once that is done, the audio is available to the Web Audio API ____
>
>     toolbox, and anything we have requirements on can be done____
>
>     __ __
>
>     3) When the processing has been done (panning, measure level, change
>     ____
>
>     level, mix) the audio would be played using an AudioDestinationNode ____
>
>     Interface
>
>
> The AudioDestinationNode represents the local client's "speakers", in
> example 3 a MediaStream is created with the line:
> var peer = context.createMediaStreamDestination();
>
> to send the processed audio to the remote peer.
>
>     ____
>
>     __ __
>
>     What is unclear to me at present, is how synchronization would work.
>     So ____
>
>     far we have been discussing in terms of that all tracks in a
>     MediaStream ____
>
>     are kept in sync; but what happens when the audio tracks are routed
>     to ____
>
>     another set of tools, and not played in the same (video) element as
>     the ____
>
>     video?
>
>
> HTMLMediaElements already have a mechanism for synchronization using the
> HTML5 MediaController API.  Live stream (live camera/mic or remote
> peers) MediaStreams would maintain synchronization (I assume you mean
> audio/video sync in this case).  The Web Audio API would just be used to
> apply effects, not changing the synchronization.
>
>     ____
>
>     __ __
>
>     Another take away is that the processing can only happen in the
>     browser ____
>
>     that is going to play the audio, since there is no way to go from an
>     ____
>
>     AudioNode to a MediaStream or MediaStreamTrack.
>
>
> There are some examples showing how to go from an AudioNode to
> MediaStream (needs to be developed for MediaStreamTrack).
> Please look especially at the use of the createMediaStreamSource()
> and createMediaStreamDestination() methods.
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html
>
>     ____
>
>     __ __
>
>     Anyone else that has looked into the Web Audio API? And any other ____
>
>     conclusions?____
>
>     __ __
>
>     I think we should give feedback from this WG (as we have some reqs
>     that ____
>
>     are relevant).____
>
>     __ __
>
>     Br,____
>
>     Stefan____
>
>     __ __
>
>     __ __
>
>     [1] http://www.w3.org/TR/2012/WD-webaudio-20120315/ ____
>
>     [2]
>     http://lists.w3.org/Archives/Public/public-webrtc/2012Mar/0072.html ____
>
>     [3] ____
>
>     http://datatracker.ietf.org/doc/draft-ietf-rtcweb-use-cases-and-requirements/?include_text=1
>     ____
>
>     __ __
>
>     Best Regards ____
>
>     __ __
>
>     James ____
>
>     __ __
>
>     __ __
>
>

Received on Friday, 30 March 2012 08:29:52 UTC