Re: Low Level Javascript API Proposal

On Wed, Oct 5, 2011 at 1:00 PM, Harald Alvestrand <harald@alvestrand.no>wrote:

> The relationship between a codec and a PeerConnection will unfortunately
> have to be rather intimate in several ways, I think.
>
> In particular, congestion control will have to have the ability to telll a
> codec to send less data, or the ability to tell a codec that it can send
> more data if it wants to. It may be possible to allow that feedback loop to
> be mediated via the Javascript layer, but I'm not enough up to speed on that
> issue to be able to say for sure yet.
>

I think there are some very powerful things you can do if such congestion
control is mediated in Javascript. Depending on the application in question
it may make sense to reduce bandwidth in different ways, trading frame rate
for frame size, or even across streams on the same connection. Only the JS
application is in a position to know what the right thing to do is, possibly
even after asking the user.

Enabling direct codec control and feedback from JS also opens up a whole
world of interesting possibilities: callbacks from the video codec with
motion information for security applications, callbacks from the audio codec
with silence detection.

Another area that concerns me with PeerConnection is that we really need to
be thinking about pluggable codecs, that the browser itself has no knowledge
about. I want to write a codec and load it using an NaCL type mechanism and
then be able to negotiate it from JS without waiting for a browser upgrade.


> The other thing that is hard to deal with in the proposed low level API is
> inter-stream relationships. If we follow the modeling path so far suggested
> of putting several MediaStream objects' video tracks down one RTP session
> and their audio tracks down another RTP session, the relationships between
> RTP sessions, SSRCs and so on can get complex fast, and may uncover several
> new ways of letting people write apps that fail in interesting ways.
>

I agree this whole area needs more thought and to be properly specified, but
I don't yet think that we need SDP to solve the problem.