Re: Requirements for Web audio APIs

On Sat, May 21, 2011 at 7:17 AM, Chris Rogers <crogers@google.com> wrote:

> On Thu, May 19, 2011 at 2:58 AM, Robert O'Callahan <robert@ocallahan.org>wrote:
>
>> My concern is that having multiple abstractions representing streams of
>> media data --- AudioNodes and Streams --- would be redundant.
>>
>
> Agreed, there's a need to look at this carefully.  It might be workable if
> there were appropriate ways to easily use them together even if they remain
> separate types of objects.  In graphics, for example, there are different
> objects such as Image, ImageData, and WebGL textures which have different
> relationships with each other.  I don't know what the right answer is, but
> there are probably various reasonable ways to approach the problem.
>

There are reasons why we need to have different kinds of image objects. For
example, a WebGL texture has to live in VRAM so couldn't have its pixel data
manipulated by JS the way an ImageData object can. Are there fundamental
reasons why AudioNodes and Streams have to be different ... why we couldn't
express the functionality of AudioNodes using Streams?

That sounds good, but I was thinking of other sorts of problems. Consider
>> for example the use-case of a <video> movie with a regular audio track, and
>> an auxiliary <audio> element referencing a commentary track, where we apply
>> an audio ducking effect to overlay the commentary over the regular audio.
>> How would you combine audio from both streams and keep everything in sync
>> (including the video), especially in the face of issues such as one of the
>> streams temporarily pausing to buffer due to a network glitch?
>>
>
> In general this sounds like a very difficult problem to solve.  Because if
> you had two <video> streams playing together, then either one of them could
> pause momentarily due to buffer underrun, so each one would have to adjust
> to the other.  Then if you had more than two, any of them could require
> adjustments in all of the others.
>

That's right. I agree it's hard, but I think we need to solve it, or at
least have a plausible plan to solve it. This is not a far-fetched use-case.

Thanks,
Rob
-- 
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]