Re: New API surface - inbound outbound streams/tracks

On 11/09/2012 01:14 PM, Adam Bergkvist wrote:
> Hi
>
> A while back I sent out a proposal [1] on API additions to represent 
> streams that are sent and received via a PeerConnection. The main goal 
> was to have a natural API surface for the new functionality we're 
> defining (e.g. Stats and DTMF). I didn't get any feedback on the list, 
> but I did get some offline.
>
> I've updated the proposal to match v4 of Travis' settings proposal [2] 
> and would like to run it via the list again.
>
> Summary of the main design goals:
> - Have a way to represent a stream instance (witch tracks) that are 
> sent (or received) over a specific PeerConnection. Specifically, if 
> the same stream is sent via several PeerConnection objects, the sent 
> stream is represented by different "outbound streams" to provide fine 
> grained control over the different transmissions.
>
> - Avoid cluttering PeerConnection with a lot of new API that really 
> belongs on stream (and track) level but isn't applicable for the local 
> only case. The representations of sent and received streams and tracks 
> (inbound and outbound) provides the more precise API surface that we 
> need for several of the APIs we're specifying right now as well as 
> future APIs of the same kind.

There are 2 reasons why I didn't like this the first time:

- "Prefer composition to inheritance" - imposing multiple levels of 
inheritance just to make sure functionality could not be reached where 
it was not appropriate did not appeal to me.
- It was not at all clear to me how the different track types turned 
into each other.

After going through the text below, I find that this still stands.
What's appealing with the model is that you create a new stream-like 
object when you connect a stream to a peerconnection, you don't just 
store a link to it. That has some advantages.

But it's an important change to the model. Don't suggest it without 
doing so explicitly - and without explaining exactly what the 
relationship between the two stream-like objects is.

For instance:

getUserMedia(...) -> callback(stream) { stream = s; }
pc.addStream(s);
s.stop();
pc.outgoingStreams.getStreamById(s.id).ended == ?

True or false?


>
> Here are the object structure (new objects are marked with *new*). 
> Find examples below.
>
> AbstractMediaStream *new*
> |
> +- MediaStream
> |   * WritableMediaStreamTrackList (audioTracks)
> |   * WritableMediaStreamTrackList (videoTracks)
> |
> +- PeerConnectionMediaStream *new*
>     // represents inbound and outbound streams (we could use
>     // separate types if more flexibility is required)
>     * MediaStreamTrackList (audioTracks)
>     * MediaStreamTrackList (videoTracks)
I don't see what's going on here. What can we do with a 
WritableMediaStreamTrackList that we can't do with a 
MediaStreamTrackList? Add streams? But we can do that to the outgoing 
stream of a PeerConnection - which is a PeerConnectionMediaStream 
according to this text.
>
> MediaStreamTrack
> |
> +- VideoStreamTrack
> |  |
> |  +- VideoDeviceTrack
> |  |   * PictureDevice
> |  |
> |  +- InboundVideoTrack *new*
> |  |   // inbound video stats
> |  |
> |  +- OutboundVideoTrack *new*
> |      // control outgoing bandwidth, priority, ...
> |      // outbound video stats
> |      // enable/disable outgoing (?)
And we can't enable/disable incoming?
> |
> +- AudioStreamTrack
>    |
>    +- AudioDeviceTrack
>    |
>    +- InboundAudioStreamTrack *new*
>    |   // receive DTMF (?)
Not clear we need this, ever.
> |   // inbound audio stats
Not clear if they're different from outgoing audio stats; if they're 
not, they should be hoisted up a level.
Does the class give enough benefit to justify its existence, or is it 
just symmetry?
> |
>    +- OutboundAudioStreamTrack *new*
>        // send DTMF
>        // control outgoing bandwidth, priority, ...
>        // outbound audio stats
>        // enable/disable outgoing (?)
>
> === Examples ===
>
> // 1. ***** Send DTMF *****
>
> pc.addStream(stream);
> // ...
Show the adding too. It's a critical piece of the type-changing.
>
> var outboundStream = pc.localStreams.getStreamById(stream.id);
So outboundStream !== stream?
> var outboundAudio = outboundStream.audioTracks[0]; // pending syntax
By contrast (or similarity), the "object oriented" model I was charged 
with producing after Lyon would have

var outboundDTMF = pc.DTMFStream(outboundStream.audioTracks[0])
>
> if (outboundAudio.canSendDTMF)
>     outboundAudio.insertTones("123", ...);
>
>
> // 2. ***** Control outgoing media with constraints *****
>
> // the way of setting constraints in this example is based on Travis'
> // proposal (v4) combined with some points from Randell's bug 18561 [3]
>
> var speakerStream; // speaker audio and video
> var slidesStream; // video of slides
>
> pc.addStream(speakerStream);
> pc.addStream(slidesStream);
> // ...
>
> var outboundSpeakerStream = pc.localStreams
>         .getStreamById(speakerStream.id);
> var speakerAudio = outboundSpeakerStream.audioTracks[0];
> var speakerVideo = outboundSpeakerStream.videoTracks[0];
>
> speakerAudio.priority.request("very-high");
> speakerAudio.bitrate.request({ "min": 30, "max": 120,
>                                "thresholdToNotify": 10 });
> speakerAudio.bitrate.onchange = speakerAudioBitrateChanged;
> speakerAudio.onconstraintserror = failureToComply;
>
> speakerVideo.priority.request("medium");
> speakerVideo.bitrate.request({ "min": 500, "max": 1000,
>                                "thresholdToNotify": 100 });
> speakerAudio.bitrate.onchange = speakerVideoBitrateChanged;
> speakerVideo.onconstraintserror = failureToComply;
>
> var outboundSlidesStream = pc.localStreams
>         .getStreamById(slidesStream.id);
> var slidesVideo = outboundSlidesStream.videoTracks[0];
>
> slidesVideo.priority.request("high");
> slidesVideo.bitrate.request({ "min": 600, "max": 800,
>                               "thresholdToNotify": 50 });
> slidesVideo.bitrate.onchange = slidesVideoBitrateChanged;
> slidesVideo.onconstraintserror = failureToComply;
This assumes a surface that includes "oncinstraintserror" on the 
OutboundMediaStream. Which level of the hierarchy does that belong to?

Furthermore, it assumes a surface called "bitrate". Is that the one 
that's unique to an OutboundMediaStream?
>
>
> // 3. ***** Enable/disable on outbound tracks *****
>
> // send same stream to two different peers
> pcA.addStream(stream);
> pcB.addStream(stream);
> // ...
>
> // retrieve the *different* outbound streams
> var streamToA = pcA.localStreams.getStreamById(stream.id);
> var streamToB = pcB.localStreams.getStreamById(stream.id);
>
> // disable video to A and disable audio to B
> streamToA.videoTracks[0].enabled = false;
> streamToA.audioTracks[0].enabled = false;
This is actually the piece that seems most appealing. It drives home the 
point that pc.addStream(stream) *creates* a stream, it does not 
*connect* one. They "just" happen to have the same ID.

>
> ======
>
> Please comment and don't hesitate to ask if things are unclear.
>
> /Adam
>
> ----
> [1] http://lists.w3.org/Archives/Public/public-webrtc/2012Sep/0025.html
> [2] 
> http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/proposals/SettingsAPI_proposal_v4.html
> [3] https://www.w3.org/Bugs/Public/show_bug.cgi?id=15861
>

Received on Friday, 9 November 2012 13:17:30 UTC