Re: RtpSender/RtpReceiver and simulcast

I like your analysis of "native simulcast" vs. "application
simulcast".  Given the Sender/Receiver split, the JS application could
always create multiple senders and send with multiple resolutions
(assuming we can control the resolutions).  But that would be somewhat
inconvenient, and the browser can probably do a better job if it knows
more about the JS's intentions.  I think I would be in favor of
including the "native simulcast" capability in the API.  Here's a very
simple version of what it could look like:

dictionary RTCRtpEncodingParameters {
  unsigned int ssrc;

  // For simulcast, have different encodings with different resolutions.
  int width;
  int height;
  double framerate;
}

With an example like so:

rtpSender.setParameters({
  codecs: [...],
  encodings: [{
      width: 1280, height: 720, framerate: 30
    }, {
      width: 640, height: 360, framerate: 30
    }, {
      width: 320, height: 180, framerate: 15
    }, {
    }
  ]
}


Now, obviously this ignores the big topic of what to do when bandwidth
is constrained (which things to drop first,
min/max/priority/whatever-complex-thing), and we'll surely need
something more refined than this, but I think it gives a basic idea of
how an RtpParameters dictionary could contain multiple encodings to
control simulcast.

And as for how the receiver tells the sender what it wants, that's up
to the application.  At the end of the day what matters is what the JS
on the sender side tells the browser to do.


On Thu, Jan 9, 2014 at 6:09 PM, Bernard Aboba
<Bernard.Aboba@microsoft.com> wrote:
> In looking at the proposed RTCRtpSender/RTCRtpReceiver split, I had
> several questions relating to the handling of encoding parameters
> (e.g. resolution, framerate, quality, etc.).
>
> My first question relates to the model for handling simulcast.
>
> Here is the relevant excerpt on encoding parameters
> (from http://lists.w3.org/Archives/Public/public-orca/2014Jan/0000.html):
>
>   // Can specify multiple multiple layers or "encodings",
>   // such as for simulcast, RTX, FEC, etc in the future.
>   sequence<RTCRtpEncodingParameters> encodings;
>
>   // TODO: RTP header extensions, "appID", ...
>
> dictionary RTCRtpEncodingParameters {
>   unsigned int ssrc;
>
>   // TODO: Things to control various layers, simulcast, rtx, etc.
>   // For now, let's just recognize that we need more than one
>   // and we need more than the SSRC.  So let's make a dictionary,
>   // and figure out the details later.
> }
>
> Looking at the above, it appears that there are two potential
> ways to handle simulcast.   One way is to utilize "application
> layer simulcast" where there are multiple RTCRtpSender
> objects, each sending a single stream, each of which
> utilizes its own ssrc and set of encoding parameters.
>
> Another way is to use "native" simulcast where a single
> RTCRtpSender object is configured to send multiple streams
> using a sequence of RTCRtpEncodingParameters.
>
> In "Application layer simulcast", several
> RTCRtpSender objects would be configured, each sending
> a different resolution.  A single RTCRtpReceiver object
> would be configured, which would be prepared to receive
> any of the potential resolutions sent by the mixer.
> In this approach, my assumption is that a "cloned"
> MediaStreamTrack would be provided as input to each of the
> RTCRtpSender objects.
>
> In "Native simulcast", a single RTCRtpSender
> object would be configured with a sequence of
> RTCRtpEncodingParameters so as to send multiple streams.
> There would also be a single RTCRtpReceiver object as
> before.
>
> In either approach the streams to be sent could be
> configured to each have their own unique SSRC.
>
> One concern for either approach is coordination of the
> resolutions to be sent.  For example, it makes no sense
> for the browser to send the same resolution on multiple
> streams.
>
> Also, to avoid disconcerting the user, it would typically
> be desirable for the resolutions sent or received to have
> a consistent aspect ratio.
>
> To provide a concrete example, let us say an application
> wishes to offer to simultaneously send resolutions of
> 640 by 360 (on PT 97) and 320 by 180 (on PT 98), as well
> as to receive either resolution (on PT 97).
>
> How would we configure the RTCRtpSender and RTCRtpReceiver
> objects to do this?
>
> For those willing to countenance the vulgarities of SDP
> below is an example O/A exchange conforming to the spirit
> of "unified plan".
>
> Warning:  This SDP O/A Exchange is rated (R)estricted, due to
> the presence of adult material.  Children Under 17 Require
> Accompanying Parent or Adult Guardian.
>
> Offer:
>
> v=0
> o=alice 2362969037 2362969040 IN IP4 192.0.2.156
> s=Offer from Simulcast Enabled Browser to Mixer
> t=0 0
> c=IN IP4 192.0.2.156
> b=AS:665
> a=group:SCR S1 S2
> m=video 49300 RTP/AVP 97
> b=AS:520
> a=rtpmap:97 H264/90000
> a=fmtp:97 profile-level-id=42c01e
> a=imageattr:97 send [x=640,y=360] recv [x=640,y=360][x=320,y=180]
> a=mid:S1
> m=video 49302 RTP/AVP 98
> a=sendonly
> a=rtpmap:98 H264/90000
> a=fmtp:98 profile-level-id=42c01e
> a=imageattr:98 send [x=320,y=180]
> a=mid:S2
>
> Answer:
>
> v=0
> o=server 823479283 1209384938 IN IP4 192.0.2.2
> s=Answer to Simulcast Enabled Client
> t=0 0
> c=IN IP4 192.0.2.43
> b=AS:665
> a=group:SCR S1 S2
> m=video 48000 RTP/AVP 97
> b=AS:520
> a=rtpmap:97 H264/90000
> a=fmtp:97 profile-level-id=42c01e
> a=imageattr:97 send [x=640,y=360][x=320,y=180] recv [x=640,y=360]

Received on Friday, 10 January 2014 19:24:08 UTC