Re: MSP stalled streams, Was: Reviewing the Web Audio API (from webrtc)

On Thu, Mar 29, 2012 at 10:50 PM, Jussi Kalliokoski <
jussi.kalliokoski@gmail.com> wrote:

> I've been thinking about this discussion about synchronization of streams
> and blocking the processing along with it, and I have a suggestion for the
> MediaStream Processing API. This is based on the idea, that it's not
> necessarily a good idea to prevent all processing if input streams are
> blocked, for example it might be good user experience to keep a reverb
> going on even if the input stream is stalled.
>

Yes, that's definitely useful. The currently specced approach is to have an
attribute on MediaInput --- blockOutput --- that control whether blocking
on the input stream requires the output stream to block. (I've also
proposed an attribute to control the other direction, but that's less
useful.) When blockOutput is false and the input stream blocks, the output
stream can still advance, and uses silence/no video for the blocked input.

This isn't implemented yet, and it's tricky to implement, so I still might
need to revisit this.

What about making the callbacks trigger regardless of whether the input
> streams are blocked, but make the event contain information about the input
> stream state, so that the developer can use her better judgement on whether
> to and which effects to keep running:
>
> partial interface MediaInputBuffer {
>     readonly attribute MediaInputState state;
> }
>
> dictionary MediaInputState {
>     active, paused, drained, blocked
> }
>
> then you could do something like this:
>
> self.onprocessmedia = function (e) {
>     if (e.inputs[0].state !== "active") return;
>     // Do something when the stream is active
> };
>

We could do something like that if effects need to be aware of why an input
stream is producing silence. Right now I can't think of why that would be
needed :-).

This approach would be useful for cases where the stream is drained as
> well, for a simplified example, there's a media file containing a gunshot,
> and the file has no silence padding at the end, so the reverb would
> overflow if the callback stops when the stream is drained.
>

In that example, there isn't really a problem. An input stream that has
finished is not an active input, and only active inputs are allowed to
block the output stream.

Thanks,
Rob
-- 
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]