Re: Reviewing the Web Audio API (from webrtc)

On Thu, Apr 19, 2012 at 10:36 AM, Robert O'Callahan <robert@ocallahan.org>wrote:

> On Wed, Apr 18, 2012 at 12:23 PM, Randell Jesup <randell-ietf@jesup.org>wrote:
>
>> So it sounds like to modify audio in a MediaStream you'll need to:
>>
>> * Extract each track from a MediaStream
>> * Turn each track into a source (might be combined with previous step)
>> * Attach each source to a graph
>> * Extract tracks from the destination of the graphs
>> * Extract the video stream(s) from the MediaStream source
>> * Combine all the tracks back into a new MediaStream
>>
>
> And one of the downsides of doing it this way is that you lose sync
> between the audio and video streams. Usually not by much, but more for
> certain kinds of processing. Given there's a way to not lose sync at all,
> why not use it? Sorry to harp on this :-).
>


Offline, Chris and I discussed some ways in which Web Audio could propagate
latency information to solve this problem. It's probably better for him to
explain his ideas since I'm not completely sure how they would work.

It gets more complicated if we have the ability to pause media streams
(which I think we will, e.g. for live broadcasts that aren't interactive,
it makes sense to pause instead of just dropping data). Since Web Audio
can't pause, the paused time has to be accounted for and essentially a time
slice of the Web Audio output corresponding to the pause interval has to be
clipped out. And that's going to be annoying if your filter is something
like an echo ... some echo would be lost, I think.

Maybe all these issues can be solved, or deemed not worth addressing, but
supporting pause seems appealing to me.

Rob
-- 
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]