Re: Reviewing the Web Audio API (from webrtc)

On Thu, Apr 26, 2012 at 3:58 PM, Robert O'Callahan <robert@ocallahan.org>wrote:

> Thanks for the overview! It's very helpful.
>
>
> On Sat, Apr 21, 2012 at 10:26 AM, Chris Rogers <crogers@google.com> wrote:
>
>> a. Audio from a <video> element is processed with latency (as in your
>> ducking example).  In this case the strategy of delaying video frame
>> presentation would be used, and could be accomplished automatically by the
>> implementation because it has all necessary information available to apply
>> this strategy.
>
>
> If I understand correctly, this means when I press "play" I have to wait
> for one second before playback starts while the AudioContext spins up.
> That's undesirable. Maybe there's another way to do it?
>

Hi Rob,

Yes, that's right given the strategy I mention of delaying video frame
presentation.  In proposing this as a possible strategy, I was actually
thinking more generally of handling video streams, potentially from a
"live" un-paused video source from WebRTC where this would be more
appropriate.  There is a strategy to avoid waiting one second when dealing
with a <video> element in the normal case where its content is available in
advance (assuming that content has been buffered).  That would be to
"prime" the processing pipeline with an amount of audio equal in duration
to the latency.  This priming can happen faster than real-time.  In the
Logic Audio link I mentioned before:
http://help.apple.com/logicpro/mac/9.1.6/en/logicpro/usermanual/index.html#chapter=41%26section=3%26tasks=true

I think this would correspond to their technique of "shifting a track
forward in time":


   -

   "If latency-inducing plug-ins are inserted into audio or instrument
   channels, Logic Pro automatically shifts these tracks forward in time.
   The advantage of this method is that other channels (that do not contain
   latency-inducing plug-ins) do not need to be delayed."



>
> I'd also appreciate your thoughts on the second paragraph of my previous
> message.
>

Sure, I think you mean your questions here:
"

> It gets more complicated if we have the ability to pause media streams
> (which I think we will, e.g. for live broadcasts that aren't interactive,
> it makes sense to pause instead of just dropping data). Since Web Audio
> can't pause, the paused time has to be accounted for and essentially a time
> slice of the Web Audio output corresponding to the pause interval has to be
> clipped out. And that's going to be annoying if your filter is something
> like an echo ... some echo would be lost, I think.
>
> Maybe all these issues can be solved, or deemed not worth addressing, but
> supporting pause seems appealing to me.
>
> "

I'm guessing you're talking about a DVR-like scenario where you can pause a
live video-stream from WebRTC?  In this case, it seems like it would make
sense to have the pause() method be in the MediaStream API very similar to
how we have a pause() method in HTMLMediaElement.  I'm not sure that the
WebRTC group has considered that, but it seems like it could be a useful
addition to their API.  If they add this method to MediaStream, then
the "shifting a track forward in time" strategy I describe above could be
used if the MediaStream has been paused, because the audio and video data
will have been buffered/stored.

If we do allow pausing a MediaStream, then we'll have to be careful to
specify what the maximum pause time would be, since the pausing would
presumably be implemented by buffering in memory for a short time, then
recording to disk.  On my DVR at home, sometimes I pause live TV and forget
about it (and go off to do something else around the house), then after
about an hour the DVR automatically un-pauses the stream since it's
buffered too much data to disk...

Cheers,
Chris





>
> Thanks,
>
> Rob
> --
> “You have heard that it was said, ‘Love your neighbor and hate your
> enemy.’ But I tell you, love your enemies and pray for those who persecute
> you, that you may be children of your Father in heaven. ... If you love
> those who love you, what reward will you get? Are not even the tax
> collectors doing that? And if you greet only your own people, what are you
> doing more than others?" [Matthew 5:43-47]
>
>