Re: MSP stalled streams, Was: Reviewing the Web Audio API (from webrtc)

On Thu, Mar 29, 2012 at 1:53 PM, Robert O'Callahan <robert@ocallahan.org>wrote:

> On Thu, Mar 29, 2012 at 10:50 PM, Jussi Kalliokoski <
> jussi.kalliokoski@gmail.com> wrote:
>
>> I've been thinking about this discussion about synchronization of streams
>> and blocking the processing along with it, and I have a suggestion for the
>> MediaStream Processing API. This is based on the idea, that it's not
>> necessarily a good idea to prevent all processing if input streams are
>> blocked, for example it might be good user experience to keep a reverb
>> going on even if the input stream is stalled.
>>
>
> Yes, that's definitely useful. The currently specced approach is to have
> an attribute on MediaInput --- blockOutput --- that control whether
> blocking on the input stream requires the output stream to block. (I've
> also proposed an attribute to control the other direction, but that's less
> useful.) When blockOutput is false and the input stream blocks, the output
> stream can still advance, and uses silence/no video for the blocked input.
>
> This isn't implemented yet, and it's tricky to implement, so I still might
> need to revisit this.
>

Oh, I managed to miss this attribute somehow. Great that it's already been
thought about!


>
> What about making the callbacks trigger regardless of whether the input
>> streams are blocked, but make the event contain information about the input
>> stream state, so that the developer can use her better judgement on whether
>> to and which effects to keep running:
>>
>> partial interface MediaInputBuffer {
>>     readonly attribute MediaInputState state;
>> }
>>
>> dictionary MediaInputState {
>>     active, paused, drained, blocked
>> }
>>
>> then you could do something like this:
>>
>> self.onprocessmedia = function (e) {
>>     if (e.inputs[0].state !== "active") return;
>>     // Do something when the stream is active
>> };
>>
>
> We could do something like that if effects need to be aware of why an
> input stream is producing silence. Right now I can't think of why that
> would be needed :-).
>

I have some use cases in mind, for example a DAW might be rendering content
to a file, let's say there's a following graph:

Media Input -> Processing Worker (reverb, etc) -> Encoder Worker

The desired behavior is that if the stream is stalled, it won't output
anything to avoid waiting for input from being written to the file, but if
the stream is drained, wait until the effects produce just silence, and
then signal the encoder to stop recording.


>
> This approach would be useful for cases where the stream is drained as
>> well, for a simplified example, there's a media file containing a gunshot,
>> and the file has no silence padding at the end, so the reverb would
>> overflow if the callback stops when the stream is drained.
>>
>
> In that example, there isn't really a problem. An input stream that has
> finished is not an active input, and only active inputs are allowed to
> block the output stream.
>

Ah, I see. Good. If that's the case, the only thing missing from my
previous use case is how to figure out when the stream has ended. This can
be checked from the main thread of course, adding event listeners to the
media elements, but it might be more convenient that the stream processor
knows about the state of the input as well.


>
> Thanks,
> Rob
> --
> “You have heard that it was said, ‘Love your neighbor and hate your
> enemy.’ But I tell you, love your enemies and pray for those who persecute
> you, that you may be children of your Father in heaven. ... If you love
> those who love you, what reward will you get? Are not even the tax
> collectors doing that? And if you greet only your own people, what are you
> doing more than others?" [Matthew 5:43-47]
>
>
Cheers,
Jussi