Re: New API surface - inbound outbound streams/tracks

Thanks for good feedback. I think this will clarify some stuff that 
didn't go into the initial mail.

On 2012-11-09 14:16, Harald Alvestrand wrote:
> On 11/09/2012 01:14 PM, Adam Bergkvist wrote:
>> Hi
>>
>> A while back I sent out a proposal [1] on API additions to represent
>> streams that are sent and received via a PeerConnection. The main goal
>> was to have a natural API surface for the new functionality we're
>> defining (e.g. Stats and DTMF). I didn't get any feedback on the list,
>> but I did get some offline.
>>
>> I've updated the proposal to match v4 of Travis' settings proposal [2]
>> and would like to run it via the list again.
>>
>> Summary of the main design goals:
>> - Have a way to represent a stream instance (witch tracks) that are
>> sent (or received) over a specific PeerConnection. Specifically, if
>> the same stream is sent via several PeerConnection objects, the sent
>> stream is represented by different "outbound streams" to provide fine
>> grained control over the different transmissions.
>>
>> - Avoid cluttering PeerConnection with a lot of new API that really
>> belongs on stream (and track) level but isn't applicable for the local
>> only case. The representations of sent and received streams and tracks
>> (inbound and outbound) provides the more precise API surface that we
>> need for several of the APIs we're specifying right now as well as
>> future APIs of the same kind.
>
> There are 2 reasons why I didn't like this the first time:
>
> - "Prefer composition to inheritance" - imposing multiple levels of
> inheritance just to make sure functionality could not be reached where
> it was not appropriate did not appeal to me.

Is it the object structure for streams or tracks (or both) that you are 
concerned about here?

Regarding the inheritance for streams (this proposal only adds new 
derived types in the track hierarchy), I wanted to have a simple stream 
type that's identifiable and playable in a media element, and then base 
the two specialized types on it. I think the "composition over 
inheritance" rule is worth considering in all cases, but I think 
inheritance is rather clean here.

It would work to simply have one type of MediaStream as well, but there 
are some benefits to have different derived types since they will be 
used a bit differently. For example, adding tracks to a stream that 
you're receiving doesn't make sense and outbound streams doesn't need it 
either since they are updated when tracks are added to the stream that 
spawned them (i.e. the stream that was added to the PeerConnection).

> - It was not at all clear to me how the different track types turned
> into each other.
>
> After going through the text below, I find that this still stands.
> What's appealing with the model is that you create a new stream-like
> object when you connect a stream to a peerconnection, you don't just
> store a link to it. That has some advantages.

Yes, it's not clear from this text; I tried to keep the size of this 
email from becoming a wall of text. :)

I think your description "stream-like object that's created when you 
connect a stream to a PeerConnection" is quite accurate. The tracks in 
the outbound stream are representations of the tracks in the stream 
added to the PeerConnection, but of the type OutboundMediaStreamTrack. 
The new type is really a decoration with functionality that's only 
related to context of the track being sent over this particular connection.

> But it's an important change to the model. Don't suggest it without
> doing so explicitly - and without explaining exactly what the
> relationship between the two stream-like objects is.

That would need to be specified, yes.

> For instance:
>
> getUserMedia(...) -> callback(stream) { stream = s; }
> pc.addStream(s);
> s.stop();
> pc.outgoingStreams.getStreamById(s.id).ended == ?
>
> True or false?

That would be true. The outbound stream is dependent on both the 
"original stream" as well as the PeerConnection it's added to (it 
represents the association between the two). If any of the two 
closes/ends, then the outbound stream will end as well.

>>
>> Here are the object structure (new objects are marked with *new*).
>> Find examples below.
>>
>> AbstractMediaStream *new*
>> |
>> +- MediaStream
>> |   * WritableMediaStreamTrackList (audioTracks)
>> |   * WritableMediaStreamTrackList (videoTracks)
>> |
>> +- PeerConnectionMediaStream *new*
>>     // represents inbound and outbound streams (we could use
>>     // separate types if more flexibility is required)
>>     * MediaStreamTrackList (audioTracks)
>>     * MediaStreamTrackList (videoTracks)
> I don't see what's going on here. What can we do with a
> WritableMediaStreamTrackList that we can't do with a
> MediaStreamTrackList? Add streams? But we can do that to the outgoing
> stream of a PeerConnection - which is a PeerConnectionMediaStream
> according to this text.

The thinking is that you shouldn't add tracks to a 
PeerConnectionMediaStream directly.

// If we do
pc.addStream(stream);
var outboundStream = pc.localStreams.getStreamById(stream.id);

// Then this is OK,
stream.audioTracks.add(newTrack);
// but not this
outboundStream.audioTracks.add(newTrack);

>>
>> MediaStreamTrack
>> |
>> +- VideoStreamTrack
>> |  |
>> |  +- VideoDeviceTrack
>> |  |   * PictureDevice
>> |  |
>> |  +- InboundVideoTrack *new*
>> |  |   // inbound video stats
>> |  |
>> |  +- OutboundVideoTrack *new*
>> |      // control outgoing bandwidth, priority, ...
>> |      // outbound video stats
>> |      // enable/disable outgoing (?)
> And we can't enable/disable incoming?

If we wanted to we could.

>> |
>> +- AudioStreamTrack
>>    |
>>    +- AudioDeviceTrack
>>    |
>>    +- InboundAudioStreamTrack *new*
>>    |   // receive DTMF (?)
> Not clear we need this, ever.

True, but that's the place to put it if we decided to.

>> |   // inbound audio stats
> Not clear if they're different from outgoing audio stats; if they're
> not, they should be hoisted up a level.
> Does the class give enough benefit to justify its existence, or is it
> just symmetry?

I don't think it's a bad think to be able to query stats (or any other 
operation) with better precision (audio/video - inbound/outbound).

>> |
>>    +- OutboundAudioStreamTrack *new*
>>        // send DTMF
>>        // control outgoing bandwidth, priority, ...
>>        // outbound audio stats
>>        // enable/disable outgoing (?)
>>
>> === Examples ===
>>
>> // 1. ***** Send DTMF *****
>>
>> pc.addStream(stream);
>> // ...
> Show the adding too. It's a critical piece of the type-changing.

Which adding do you refer to? The outbound stream is automatically 
created when the stream is added; you don't have to add it to 
pc.localStreams manually.

>>
>> var outboundStream = pc.localStreams.getStreamById(stream.id);
> So outboundStream !== stream?

False; they are not of the same type (MediaStream vs. 
PeerConnectionMediaStream), but outboundStream represents the same 
underlying sources as stream. The things you do on stream are reflected 
on outboundStream, but you can tweak things on outboundStream that's 
only valid with this particular PeerConnection instance.

>> var outboundAudio = outboundStream.audioTracks[0]; // pending syntax
> By contrast (or similarity), the "object oriented" model I was charged
> with producing after Lyon would have
>
> var outboundDTMF = pc.DTMFStream(outboundStream.audioTracks[0])

Yes, that would be quite similar. A DTMFStream would contain a subset of 
the functionality of an OutboundAudioStreamTrack.

>>
>> if (outboundAudio.canSendDTMF)
>>     outboundAudio.insertTones("123", ...);
>>
>>
>> // 2. ***** Control outgoing media with constraints *****
>>
>> // the way of setting constraints in this example is based on Travis'
>> // proposal (v4) combined with some points from Randell's bug 18561 [3]
>>
>> var speakerStream; // speaker audio and video
>> var slidesStream; // video of slides
>>
>> pc.addStream(speakerStream);
>> pc.addStream(slidesStream);
>> // ...
>>
>> var outboundSpeakerStream = pc.localStreams
>>         .getStreamById(speakerStream.id);
>> var speakerAudio = outboundSpeakerStream.audioTracks[0];
>> var speakerVideo = outboundSpeakerStream.videoTracks[0];
>>
>> speakerAudio.priority.request("very-high");
>> speakerAudio.bitrate.request({ "min": 30, "max": 120,
>>                                "thresholdToNotify": 10 });
>> speakerAudio.bitrate.onchange = speakerAudioBitrateChanged;
>> speakerAudio.onconstraintserror = failureToComply;
>>
>> speakerVideo.priority.request("medium");
>> speakerVideo.bitrate.request({ "min": 500, "max": 1000,
>>                                "thresholdToNotify": 100 });
>> speakerAudio.bitrate.onchange = speakerVideoBitrateChanged;
>> speakerVideo.onconstraintserror = failureToComply;
>>
>> var outboundSlidesStream = pc.localStreams
>>         .getStreamById(slidesStream.id);
>> var slidesVideo = outboundSlidesStream.videoTracks[0];
>>
>> slidesVideo.priority.request("high");
>> slidesVideo.bitrate.request({ "min": 600, "max": 800,
>>                               "thresholdToNotify": 50 });
>> slidesVideo.bitrate.onchange = slidesVideoBitrateChanged;
>> slidesVideo.onconstraintserror = failureToComply;
> This assumes a surface that includes "oncinstraintserror" on the
> OutboundMediaStream. Which level of the hierarchy does that belong to?
>
> Furthermore, it assumes a surface called "bitrate". Is that the one
> that's unique to an OutboundMediaStream?

Yes.

This is modeled after DeviceTracks from Travis' proposal, but instead of 
having settings/constraints on local devices it exposes the same for 
network related things.

>>
>>
>> // 3. ***** Enable/disable on outbound tracks *****
>>
>> // send same stream to two different peers
>> pcA.addStream(stream);
>> pcB.addStream(stream);
>> // ...
>>
>> // retrieve the *different* outbound streams
>> var streamToA = pcA.localStreams.getStreamById(stream.id);
>> var streamToB = pcB.localStreams.getStreamById(stream.id);
>>
>> // disable video to A and disable audio to B
>> streamToA.videoTracks[0].enabled = false;
>> streamToA.audioTracks[0].enabled = false;
> This is actually the piece that seems most appealing. It drives home the
> point that pc.addStream(stream) *creates* a stream, it does not
> *connect* one. They "just" happen to have the same ID.

Yes, it creates a new stream that represents the original stream 
associated with this PeerConnection.

/Adam

Received on Monday, 12 November 2012 11:10:36 UTC