Re: Adding Web Audio API Spec to W3C Repository

On Sat, Jun 11, 2011 at 8:42 AM, Joseph Berkovitz <joe@noteflight.com>wrote:

> After looking at the RTCStream API I find that I have an initial bias
> (which could change easily). The starting bias is this: streams seem to be
> good abstractions for setting up session-level graphs of cameras,
> microphones and remote participants where data needs to be routed and mixed,
> but not so much processed, triggered or synthesized. Streams may also be on
> the heavy side for slinging local buffers of audio sample frames around with
> some processing. So I wonder if it is better to take an entire audio
> processing graph and treat this as a single object participating as a Stream
> (source and/or sink) in an RTC session; inside that graph, there may be
> implementation economies that are unique to audio nodes which are not as
> easy to achieve when every node can also be an arbitrarily time-coded,
> interruptible Stream.
>

As an implementor, I can't think of any.

Also, if you're going to allow two-way bridging between Streams and
AudioNodes, and you're going to maintain synchronization over the entire
super-graph --- and you're going to integrate cleanly with HTML media
elements, and proposed HTML MediaControllers, and goodness knows what else
--- then it seems to me the sanest way to implement that, by far, is to have
a single internal stream graph which all those features map onto. You could
still identify optimization opportunities dynamically and use different
kinds of streams internally if you needed to.

By the way, do streams help with MIDI or with controller-type data? I don't
> see that they do, yet. Neither proposal seems to do a good job in this area
> yet.
>

As currently specced, Streams contain audio and video, but they could be
cleanly extended to contain other kinds of timed media as well --- I was
thinking Kinect-style depth information. Controller data might make sense
too.

Rob
-- 
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]