PeerConnection representation of MediaStreams that are sent/received

Hi

A while back there was a proposal on the media capture TF list [1] to 
remove the "enabled" attribute from MediaStreamTrack and let the stream 
consumer select which track(s) to treat as active. The media elements 
supports this via the enabled (audio) and selected (video) properties on 
its internal representation of the tracks from the media source. A 
consequence of this is that PeerConnection (as a stream consumer) must 
take the responsibility to control which tracks that are sent over the 
network. This proposal attempts to address this problem as well as other 
issues we currently have where it's not obvious where APIs should be 
attached (e.g. DTMF and stats).

The core of the proposal is to introduce two new MediaStream types (with 
corresponding track types) that represents streams under the control of 
PeerConnection:
- OutboundMediaStream: represents a stream that is being sent
- InboundMediaStream: represents a stream that is being received

The impact this will have on the current API is that a stream added to a 
PeerConnection will get an additional representation that is associated 
with the PeerConnection instance the stream was added to. Adding the 
same stream to two different PeerConnection instances will give you two 
different over-the-wire-representations (OutboundMediaStream) of that 
stream. The upside to this is that we have a new point to attach APIs 
that are not meaningful in the local-only use case, but would work best 
being associated with a stream instance. The downside is a greater 
number of objects involved, but I think it will be rather logical from 
the context which functionality an object supports. More objects also 
gives us the ability to, e.g., not expose an API for adding tracks to a 
stream that we're receiving from another peer (InboundMediaStream).

Issues that this design helps to solve:

* Stream consumer enable/disable (mentioned above):
We want control on track level for tracks sent with PeerConnection. The 
place for this API is therefore OutboundMediaStreamTrack.

* Stats API:
Only valid together with PeerConnection and should therefore be on 
PeerConnection's representation of streams/tracks. APIs for collecting 
stats related to sending and receiving can be split between 
OutboundMediaStream/OutboundMediaStreamTrack and 
InboundMediaStream/InboundMediaStreamTrack respectively.

* DTMF API:
Only valid together with PeerConnection and should therefore be on 
PeerConnection's representation of streams/tracks. APIs for sending DTMF 
and receiving DTMF can be split between 
OutboundMediaStream/OutboundMediaStreamTrack and 
InboundMediaStream/InboundMediaStreamTrack respectively.

* Error reporting:
Errors related to sending and receiving streams/track can fire events on 
the PeerConnection representation of these objects to give more precise 
error reporting.

* When is media transmitted?:
I've heard people requesting this kind of functionality. Similar to 
error reporting, PeerConnection's stream/track representation can be 
used as target for an event that's fired when media starts flowing.


General pros:

* The different stream/track types have clear responsibilities when it 
comes to being network oriented or for the local-only use case. This 
helps us divide them between the webrtc and getusermedia specs.

* The additions to the API does not really make it harder to use. The 
simplest case would not come in contact with the new objects.

This proposal is intended to be compatible with our current API [2] as 
well as with Travis' proposal on best-effort getUserMedia() [3].

Example idls are found below. I know there's a lot of new interfaces, 
but these will never be visible to the developer in practice. It will 
appear to the developer as if a MediaStream has different capabilities 
in different contexts.

Feedback and comments are welcome.

/Adam

[1] 
http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0029.html
[2] http://dev.w3.org/2011/webrtc/editor/webrtc.html
[3] 
http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0143.html

// *** Changes to PeerConnection

interface PeerConnection : EventTarget {
     ...
     // replaces localStreams and remoteStreams
     readonly attribute OutboundMediaStream[] outboundStreams;
     readonly attribute InboundMediaStream[] inboundStreams;
     ...
}

[Constructor(DOMString type, InboundMediaStreamEventInit eventInitDict)]
interface InboundMediaStreamEvent : Event {
     readonly attribute InboundMediaStream? stream;
};
dictionary InboundMediaStreamEventInit : EventInit {
     InboundMediaStream stream;
};


// *** Stream Types ***

interface AbstractMediaStream {
     // abstract class that acts as the base for all stream types

     // playable with media elements

     readonly attribute DOMString label;
              attribute boolean ended;
              attribute EventHandler onended;
};

interface LocalMediaStream : AbstractMediaStream {
     // LocalMediaStream should also be an AbstractMediaStream

     // TBD (see [3])
     // alternatives are:
     // * AudioDevice and VideoDevice attributes
     // * MediaStreamTrackList objects (containing LocalMediaStreamTrack
     //   objects)

     void stop();
};

[Constructor (TracksUnionType? trackContainers)]
interface MediaStream : AbstractMediaStream {
     readonly attribute WritableMediaStreamTrackList audioTracks;
     readonly attribute WritableMediaStreamTrackList videoTracks;
};

// PeerConnection's representation of streams that it's sending and
// receiving.

interface OutboundMediaStream : AbstractMediaStream {
     readonly attribute MediaStreamTrackList audioTracks;
     readonly attribute MediaStreamTrackList videoTracks;

     // extended API
     // - statistics (sending) on stream level?
     // - errors (sending) on stream level?
};

interface InboundMediaStream : AbstractMediaStream {
     readonly attribute MediaStreamTrackList audioTracks;
     readonly attribute MediaStreamTrackList videoTracks;

     // extended API
     // - statistics (receiving) on stream level?
     // - errors (receiving) on stream level?
};


// *** Track List Types ***

interface MediaStreamTrackList {
     readonly attribute unsigned long length;
     MediaStreamTrack item (unsigned long index);

              attribute EventHandler onaddtrack;
              attribute EventHandler onremovetrack;
};

interface WritableMediaStreamTrackList : MediaStreamTrackList {
     void add (MediaStreamTrack track);
     void remove (MediaStreamTrack track);
};


// *** Track Types ***

interface MediaStreamTrack {
     readonly attribute DOMString kind;
     readonly attribute DOMString label;

     // enable/disable functionality is moved to consumer

     readonly attribute MediaStreamTrackState readyState;
              attribute EventHandler onmute;
              attribute EventHandler onunmute;
              attribute EventHandler onended;
};

( interface LocalMediaStreamTrack : MediaStreamTrack { )
(     // API to control local devices                  )
( };                                                   )

// PeerConnection's representation of tracks that it's sending and
// receiving.

interface OutboundMediaStreamTrack : MediaStreamTrack {
     // consumer enable/disable
              attribute boolean enabled;

     // DTMF (send)
     readonly attribute boolean canSendDTMF;
     void sendDTMF(DOMString tones, unsigned short duration);

     // extended API
     // - statistics (sending) on track level
     // - errors (sending) on track level
};

interface InboundMediaStreamTrack : MediaStreamTrack {
     // DTMF (receive)
              attribute EventHandler ontone;

     // extended API
     // - statistics (receiving) on track level
     // - errors (receiving) on track level
};

Received on Tuesday, 4 September 2012 12:15:49 UTC