Re: Low Level Javascript API Proposal

On 05/10/2011 13:00, Harald Alvestrand wrote:
> In particular, congestion control will have to have the ability to telll
> a codec to send less data, or the ability to tell a codec that it can
> send more data if it wants to. It may be possible to allow that feedback
> loop to be mediated via the Javascript layer, but I'm not enough up to
> speed on that issue to be able to say for sure yet.

I think there are some very powerful things you can do if such 
congestion control is mediated in Javascript. Depending on the 
application in question it may make sense to reduce bandwidth in 
different ways, trading frame rate for frame size, or even across 
streams on the same connection. Only the JS application is in a position 
to know what the right thing to do is, possibly even after asking the user.

Enabling direct codec control and feedback from JS also opens up a whole 
world of interesting possibilities: callbacks from the video codec with 
motion information for security applications, callbacks from the audio 
codec with silence detection.

Another area that concerns me with PeerConnection is that we really need 
to be thinking about pluggable codecs, that the browser itself has no 
knowledge about. I want to write a codec and load it using an NaCL type 
mechanism and then be able to negotiate it from JS without waiting for a 
browser upgrade.

> The other thing that is hard to deal with in the proposed low level API
> is inter-stream relationships. If we follow the modeling path so far
> suggested of putting several MediaStream objects' video tracks down one
> RTP session and their audio tracks down another RTP session, the
> relationships between RTP sessions, SSRCs and so on can get complex
> fast, and may uncover several new ways of letting people write apps that
> fail in interesting ways.

I agree this whole area needs more thought and to be properly specified. 
Our intention was to signal this information through the syncGroup hint, 
which I'm told is a similar mechanism to that used by QuickTime.

Neil

Received on Wednesday, 5 October 2011 12:44:27 UTC