Re: Proposal-ORTC Sender / Receiver Capabilities Based Model

FYI, since this is very big, it will take me a while to read through.  Plus
I'm out all next week on vacation, and then the IETF/W3C interim is the
week after.  So I might not have a good response for a few weeks.


On Wed, May 7, 2014 at 9:13 PM, Robin Raymond <robin@hookflash.com> wrote:

>
> I am contributing a proposal on how to resolve an issue discovered in the
> usage of "parameters". While the details can always be tweaked, I think it
> successfully resolves much of the concern around the level and knowledge
> required to configure a "parameters" object for anything other than the
> basic use cases.
>
> In response to this posting:
> http://lists.w3.org/Archives/Public/public-ortc/2014May/0007.html
>
> This also addresses the issue of exchanging detailed parameters over the
> wire and instead base parameters based on capabilities.
>
> I am going to copy the entire proposal below to official contribute the
> proposal but for the sake of readability I am also including a link to the
> google doc(s).
>
> Proposal-ORTC Sender / Receiver Capabilities Based Model
>
> https://docs.google.com/document/d/1htyRaNjXTE_O1GhD8TcLCNXFvVsgszpE8Lqgp3OCHlU/edit?usp=sharing
>
> Proposal-ORTC Sender / Receiver Use Case [Usage Comparison Analysis]
>
> https://docs.google.com/document/d/1hdhCHj-gpwv06vIbAftxMG3oZtz7A-nuYsuwQEkTat4/edit?usp=sharing
>
>
>
>
> Proposal-ORTC Sender / Receiver Capabilities Based Model
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Introduction After attempting to write out some use cases using the
> existing RTCRtpSender and RTCRtpReciever objects and parameters for ORTC,
> some issues were discovered. Specifically, the application developer would
> need to have a fair amount of knowledge on exactly how to tweak low level
> parameters for anything beyond very simple use cases. For example, setting
> up an SVC (Scalable Video Codec) would have required knowing about what
> codecs support SVC, how the layering is setup for particular codecs, and
> finally setting up specific geometric (or temporal) attributes and layering
> relationship details by an application developer.As a result of the lack of
> easily configuration of RTP features, the idea came out to give the
> application developer "preferences" where the developer could choose what
> they want desire with high level knobs and dials and let the engine (which
> has explicit knowledge of each codec) configure the low level "parameters"
> details according to a developer's wishes. The engine could then return the
> closest set of preferences that could be achieved given the capabilities of
> the engine and the developer can then choose to proceed or not setting up
> media flows using these preferences and constructed parameters.Another
> important discovery was made in the process of defining "preferences". If
> two ORTC engines were given the same set of preferences and the
> capabilities of both sender and receiver, each engine could be made to
> construct "compatible" sender and receiver "parameters" details without
> ever exchanging the parameter details over the wire. This small realization
> about generating "parameters" from capabilities for local consumption by an
> engine has a huge impact. This generation removes the need for an engine to
> understand and filter settings that it may not understand created by
> another engine of unknown origin, which may use proprietary and/or custom
> settings. A simple "ignore capabilities you don't understand" rule could
> replace complex and cumbersome rules that would be otherwise required if
> "parameters" were to be sent over the wire and later filtered using a set
> of capabilities.Parameters can be generated based on the union of sender
> and receiver capabilities along with application developer preferences
> being used as a guideline on how to create the parameters. The engine will
> do it’s best to fulfill the preferences and it will return the parameters
> that are possible given the union of the capabilities.Two different engines
> must be able to compute compatible parameters given all the same
> preferences and capabilities. Fortunately, any two engines that understand
> the same capabilities can easily follow the same rules to generate
> compatible parameters. While the parameters created on the sender and
> receiver are required to be "compatible", they need not be identical. The
> application developer should call "createParameters(...)" on sender to
> create parameters suitable for the sender. The application developer should
> call "createParameters(...)" on the receiver to create params suitable for
> a receiver. The calculated “parameters” for both sender and receiver have
> to be compatible only to the extent that whatever a sender produces a
> receiver must be capable of decoding.The application developer has the
> option to tweak the detailed parameters output by "createParameters(...)"
> but should only do so with extreme caution. The resultant parameters output
> by "createParameters(...)" are only meant for local consumption by the
> local sender / receiver “start” methods.  Sending these created parameters
> over the wire is discouraged because implementations may produce objects
> which may not be entirely understable by the remote party, even though the
> media sent on the wire will be compatible. Differences from Current Sender
> / Receiver APIBoth models and APIs are more similar than they are
> different. The subtle differences make important behavioural usage
> implications.Both models send and receive based upon "parameter" settings.
> The difference is in how the "parameters" are generated. The new model
> generates the "parameters" based on an exchange of capabilities and the
> application developer is given convenient 'knobs' called "preferences" to
> perform most common use cases. The "parameters" in the new model are
> intended for local consumption only and the application developer is not
> required (and actively discouraged) from marshalling these "parameters"
> over the wire. The new model proposes marshaling and exchanging
> "capabilities" and optionally "preferences" and then generating compatible
> "parameters" based on those exchanges.In both models, the application
> developer may choose to tweak low level parameters should specific
> compatibilities be required. But the "preferences" model allows most
> application developers to completely ignore the low level
> parameters.Advantages of the New Capabilities ModelOverall the proposed
> capabilities based API has strong advantages. Main advantages are: 1.
> Simplicity in setup based on "preferences" for the application developer2.
> Less brittle designs/implementations since low level parameters are not
> exchanged, filtered, and interpreted by different browser engines3. Much
> less knowledge (and often no pre-knowledge) is required for the application
> developer to take full advantage of a browser's capabilities RTCRtpSender /
> RTCRtpReceiverinterface RTCRtpSender {  // ...  static RTCRtpParameters
> createParameters(    MediaStreamTrack track,    Capabilities receiverCaps,
>    optional (RTCRtpAudioPreferences or              RTCRtpVideoPreferences
> or              RTCRtpSimulcastPreferences) prefs,    optional Capabilities
> senderCaps   // optional as system can obtain this information  );  void
> start(RTCRtpParameters params);  // ...); interface RTCRtpReceiver {  //
> ...  static RTCRtpParameters createParameters(    DOMString kind,
>    Capabilities senderCaps,    optional (RTCRtpAudioPreferences or
>              RTCRtpVideoPreferences or
>              RTCRtpSimulcastPreferences) prefs,    optional Capabilities
> receiverCaps // optional as system can obtain this information  );  void
> start(RTCRtpParameters params);  //...); RTCRtpMediaPreferences// This is
> the base dictionary used for both audio and video preferences and
> represents// the set of common preferences that are available for both
> media types.dictionary RTCRtpMediaPreferences {    // If not specified,
> system will choose value. If specified, this receiverId will    // be
> applied to primary SSRC “as is”. If more than one SSRC is needed to encode
>    // the stream (e.g. FEC, RTX, MST, simulcast), where the meaning of the
> RTP packet    // with that alternative SSRC cannot be determined by the
> media flow itself, the    // alternative SSRCs will construct a receiverId
> value based upon this receiverId    // value.    DOMString
>            receiverId;    // This is the primary SSRC to use. Should
> alternative SSRCs be required (e.g. FEC,    // RTX, MST, simulcast), all
> other SSRCs should be assigned sequentially starting    // from the chosen
> SSRC value.    unsigned int         ssrc;    // For a sender, force the
> chosen codec to be the codec within the RTCRtpCapabilities    // with this
> name. If possible to choose this codec, the system will confirm by    //
> choosing this codec in the result from "createParameters(...)".    // This
> value has no meaning for a receiver since a receiver must be capable    //
> of receiving any of the compatible codecs within the union
> RTCRtpCapabilities.    // A non specified value indicates the system will
> choose the preferred sending    // codec.    DOMString
>            codecName;   // This value indicates the relative importance of
> the media being sent with a   // sender versus other media being sent. The
> logic is that all sent media with   // the same priority will be treated as
> having an equal priority. Those with   // a greater value will be given a
> greater priority and those with a lower value   // will be given a lower
> priority. The value is relative meaning a value of 2.0   // should be given
> roughly 2 times the priority vs a 1.0 value and a value of 4.0   // should
> be given roughly 4 times the priority vs a 1.0 value.   double
>                relativePriority = 1.0;   // This value indicates the
> maximum bit rate the media is allowed to output as   // a combined whole
> (including all layers, FEC, RTX, etc). The system will filter   // out
> codecs that are not capable of delivering below this bit rate unless no
>   // codec is possible in which case the system will chose the minimal
> codec bit rate   // possible and will override with a different maximum bit
> rate in the result of   // "createParameters(...)".   double
>                maxBitrate;              // engine, keep under this rate
>   // These values indicates the preferred treatment of FEC/RTX for the RTP
> packets. For   // audio, some audio codecs have built in FEC/RTX mechanisms
> in which case if the   // codec is capable, the codec should enable its
> FEC/RTX mode if value is set to all   // for that codec rather than
> creating an additional RTP flow.   RTCRtpRecoveryOptions fec = "none";
>   RTCRtpRecoveryOptions rtx = "none";}; enum RTCRtpRecoveryOptions {
>   "all",     // apply to all layers   "base",    // only apply for base
> (audio will treat "base" as equivalent to "all")   "none"     // do not
> apply to any layer}; RTCRtpAudioPreferencesdictionary
> RTCRtpAudioPreferences : RTCRtpMediaPreferences {   // If not 0, tells the
> engine to pick and configure codecs that are capable of   // the minimum of
> channels (if possible). If not possible, the minimum number of   //
> channels will be returned in the result of "createParameters(...)".
>   unsigned int         minChannels = 0;   // If not 0, tells the engine to
> pick a codec and configure codecs which are   // capable of delivering the
> minimum Hz rate as indicated. If not possible, the   // minimum Hz rate
> will be returned in the result of "createParameters(...)"   unsigned int
>         minHzRate = 0;   // The engine will choose and configure the codecs
> best able to deliver the level   // of fidelity requested.
>   RTCRtpAudioFidelity  fidelity = "speech";}; enum RTCRtpAudioFidelity {
>  "speech",   // speech only is expected so Hz range only need to support
> the vocal range  "music",    // music is expected, choose stereo compatible
> and minimal 32000 Hz  "movie"     // music / sound effects expected, choose
> surround and highest Hz available }; RTCRtpVideoPreferences (and
> related)dictionary RTCRtpVideoPreferences : RTCRtpMediaPreferences {   //
> minFrameRate, minScale, and minQuality each indicate that the engine must
> do   // it's best effort to keep the frame rate, scale or quality above a
> certain minimal   // level. When using SVC, these values will hint at the
> requirements typically needed   // for the base layer.   //   //
> minFrameRate is specified in frames per second.   double     minFrameRate =
> 0;    // please engine, keep equal or above this rate   // minScale is a
> relative value from 0.0 to 1.0 where 1.0 represents full input stream   //
> width/height is requested and 0.0 represents no minimize size is requested.
>   // The value of minScale is multiplied by the source video window width
> and height   // to calculate a minimal width and height that is relative to
> source size.   double     minScale = 0;        // please engine, keep equal
> or above this scale   // Alternatively, a specific fixed minimal width and
> height can be requested.   double     minWidth = 0;        // please
> engine, keep above X pixels wide   double     minHeight = 0;       //
> please engine, keep above Y pixels high   // minQuality is a relative value
> from 0.0 to 1.0 where 1.0 means maximum output   // quality is requested
> for a given codec and 0.0 allows any minimal codec quality   // output is
> deemed acceptable.   double     minQuality = 0;      // please engine, keep
> equal or above this quality   // The engine needs values to help decide
> what to sacrifice when network conditions   // are not ideal. The
> frameRatePriority, scalePriority, and qualityPriority indicate   // the
> relative importance of each aspect of the video relative to the other (or
>   // 0.0 which means the video aspect has no significance (with exclusion
> to the minimum   // above). The values are relative to each other thus a
> value of 2.0 vs 1.0 has   // roughly 2 times the importance and a value of
> 4.0 vs 1.0 has roughly 4 times the   // importance (relatively speaking).
>   double    frameRatePriority = 1.0; // priority of frame rate   double
>    scalePriority = 1.0;     // priority of scale   double
>    qualityPriority = 1.0;   // priority of quality   // If a type of SVC
> layering is desired, the frameRateScalabilityOptions,   //
> scalingScalabilityOptions, and qualityScalabilityOptions should be set to a
>   // non-null value for each SCV type desired. The details of the   //
> RTCRtpScalabilityOptions dictionary will indicate the desired details for
>   // each individual SVC type requested.   //   // Default of null
> indicates no SVC of specific type is requested.   RTCRtpScalabilityOptions?
> frameRateScalabilityOptions = null;   RTCRtpScalabilityOptions?
> scalingScalabilityOptions = null;   RTCRtpScalabilityOptions?
> qualityScalabilityOptions = null;}; dictionary RTCRtpScalabilityOptions {
>   // If the alternative value other than the default value of null is
> specified, this   // indicates to the engine the precise number of layers
> desired (if possible for a   // given codec to deliver these layers). If
> null, the engine is free to choose   // the default layering statically or
> dynamically dependent upon the codec   // capabilities.   unsigned int?
>     layers = null;}; RTCRtpSimulcastPreferencesdictionary
> RTCRtpSimulcastPreferences {   // This value indicates the maximum bit rate
> all media is allowed to output as   // a combined for all simulcast
> streams.   double?               maxBitrate = null;           // engine,
> keep under this rate   sequence<RTCRtpVideoPreferences> simulcastStreams;};
> RTCRtpParameters// Typically this object is constructed by the RTCRtpSender
> for local consumption by// the RTCRtpSender and by the RTCRtpReceiver for
> local consumption by a RTCRtpReceiver. // This is a "shotgun" object,
> meaning the developer is given the power of a "shotgun"// pointed at their
> feet and they can mess with this object at their own peril should// they
> need to modify it for unusual compatibility reasons. Normal use cases
> should not// require modifying the values within this structure and
> marshalling this structure for// remote consumption by another browser
> engine is highly discouraged.dictionary RTCRtpParameters {   // When
> returned as a result, the system will express the actual chosen preferences
>   // possible to best fulfill the preferences given the capabilities. In
> other words,   // the developer can't always get what they want; but if
> they try sometimes, they will   // get what they need.
>   (RTCRtpAudioPreferences or    RTCRtpVideoPreferences or
>    RTCRtpSimulcastPreferences) preferences;   // the capabilities of both
> sender and receiver [value "as is" when passed   //
> "createParameters(...)]"   RTCRtcCapabilities senderCapabilities;
>   RTCRtcCapabilities receiverCapabilities;   // This value contains all the
> particularly low level details of how the engine   // will encode the media
> on the wire.   (RTCRtpParameterAudioDetails or
>    RTCRtpParameterVideoDetails or    RTCRtpParameterSimulcastDetails)
>  details;   // The chosen RTP features based upon the union of the
> capabilities.   Settings rtpFeatures;   // The chosen RTP extensions and
> configurations based upon the union of   // the capabilities.
>   sequence<RTCRtpHeaderExtensionParameters>? headerExtensions = null;};
> RTCRtpParameterDetails// This is the base dictionary of common parameters
> needed for both audio and video media// types. Audio and video will each
> have their own set of specific parameters depending// upon the media
> type.dictionary RTCRtpParameterDetails {   DOMString        receiverId =
> "";    // use this receiver ID for RTP stream ("" = N/A)   unsigned int
>     ssrc = null;        // using this SSRC for RTP stream   DOMString
>        fecReceiverId = ""; // use this receiver ID for FEC RTP ("" = N/A)
>   unsigned int?    fecSsrc = null;     // using this SSRC for FEC (null =
> N/A)   Settings         fec;                // modes of operation related
> to FEC   DOMString        rtxReceiverId = ""; // use this receiver ID for
> RTX RTP ("" = N/A)   unsigned int?    rtxSsrc = null;     // using this
> SSRC for FEC (null = N/A)   Settings         rtx;                // modes
> of operation related to RTX   // null for a sender. For a receiver, this
> must contain the source SSRC to   // use for RTCP Receiver Reports (RRs).
>   unsigned int?    rtcpSsrc = null;   // If true, the engine will mux RTCP
> with RTP on the same RTCIceTransport. If false,   // the engine will send
> RTCP reports on the associated RTCP RTCIceTransport component.   boolean
>          rtcpMux = true;}; RTCRtpParameterAudioDetails (and
> related)dictionary RTCRtpParameterAudioDetails : RTCRtpParameterDetails {
>   // Contains a list of audio codec options per possible to use codecs. The
> order   // of the codecs is in preferred order.
>   sequence<RTCRtpParameterAudioCodecDetails> codecDetails;}; dictionary
> RTCRtpParameterCodecDetails {   // The name of the codec as related to the
> codec name(s) contained within the codecs   // listed within the
> RTCRtpCapabilities dictionaries.   DOMString       codecName;   unsigned
> byte   payloadType;       // actual payload type sent on wire   Settings
>        formatsParameters; // detailed settings chosen for related codec};
> dictionary RTCRtpParameterAudioCodecDetails : RTCRtpParameterCodecDetails {
>   // nothing anything required at this time?}; RTCRtpParameterVideoDetails
> (and related)dictionary RTCRtpParameterVideoDetails :
> RTCRtpParameterDetails {   double           scale = 1.0;        // 0..1
> relative scale from source   double           frameRate = 1.0;    // 0..1
> relative frame rate from source   double           quality = 1.0;      //
> 0..1 relative quality from source   // Contains a list of video codec
> options per possible to use codecs. The order   // of the codecs is in
> preferred order.   sequence<RTCRtpParameterVideoCodecDetails>
> codecDetails;}; dictionary RTCRtpParameterVideoCodecDetails :
> RTCRtpParameterCodecDetails {   // When layering is used, this value
> contains a sequence containing the layer   // information as needed for the
> related codec.   sequence<RTCRtpParameterVideoLayerDetails>? layers =
> null;}; dictionary RTCRtpParameterVideoLayerDetails {   // Value is set if
> required for describing the dependency tree information for   // the
> codec's layers.   DOMString              layerId = "";   // Value is null
> for the base layer or if dependencies are not needed to be   // described
> (as may be the case for dynamic SCV codecs). If set, the value   //
> contains a list of layers this layer is dependent upon (thus allowing a
>   // dependency tree/graph to be created).   sequence<DOMString>?
>   layerIdDependencies = null;   RTCRtpScalabilityType? layerScalabilityType
> = null;  // null would be for base   DOMString              receiverId =
> "";  // use this receiver ID in layer ("" = N/A)   unsigned int?
>          ssrc = null;      // if layer uses its own SSRC (null = N/A)
>   double?                frameRate = null; // framerate for layer (for
> temporal SVC)   double?                scale = null;     // scale applied
> to layer (for spatial SVC)   double?                quality = null;   //
> quality applied to layer (for quality SVC)   DOMString
>              fecReceiverId = ""; // receiver ID for FEC RTP ("" = N/A)
>   unsigned int?          fecSsrc = null;     // using this SSRC for FEC
> (null = N/A)   Settings               fec;                // modes of
> operation related to FEC   DOMString              rtxReceiverId = ""; //
> receiver ID for RTX RTP ("" = N/A)   unsigned int?          rtxSsrc = null;
>     // using this SSRC for FEC (null = N/A)   Settings               rtx;
>                // modes of operation related to RTX}; enum
> RTCRtpScalabilityType {   "temporal",   "spatial",   "quality"};
> RTCRtpParameterSimulcastDetails (and related)dictionary
> RTCRtpParameterSimulcastDetails {   // This sequence contains the details
> of each simulcasted stream when simulcasting   // is used or will contain
> exactly 1 video stream details when not simulcasting.
>   sequence<RTCRtpParameterVideoDetails>? simulcastStreams;}; RTCRtpCodec
> Dictionary Tweakdictionary RTCRtpCodec {    DOMString     name = "";    //
> Added to be able to pick payload type based upon sender or receiver so they
> match    // when creating both the sender and receiver parameters.
>    unsigned byte preferredPayloadType;    unsigned int? clockRate = null;
>    unsigned int? numChannels = 1;    Capabilities  formats;}; *Proposal-ORTC
> Sender / Receiver Use Case [Usage Comparison Analysis]
>
> Introduction
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *After attempting to work through examples of code usage using the current
> ORTC sender/receiver API, some issues, concerns and deficiencies were
> discovered. A retuning of the current model was made to attempt to address
> those findings. The differences are illustrated below in code examples
> based on various use cases.In the first set of use cases for simple
> application usages, there are no advantages to a capabilities model (aside
> from the reduction of complexity an engine might need to implement). As the
> use cases become more involved, advantages begin to show. In the final
> example which illustrates using SVC, the clear advantage of capabilities
> and preferences can be demonstrated. Use CasesAlice wishes to send media to
> BobCurrent Parameter Based APIStep 1: (Alice)var track =
> myObtainMediaTrack();var senderCaps = RTCRtpSender.getCapabilities();var
> senderParams = RTCRtpSender.createParameters(track,
> senderCaps);mysignal(senderParams); Step 2: (Bob)var senderParams =
> mysignal();var receiverCaps = RTPRtcReceiver.getCapabilities();var
> receiverParams = RTPRtcReceiver.filterParameters(senderParams,
> receiverCaps);var receiver = new
> RTCRtpReceiver(...);receiver.start(receiverParams);
> mysignal(receiverParams); Step 3: (Alice)var receiverParams =
> mysignal();var senderParams = RTPRtcSender.filterParameters(receiverParams,
> senderCaps);var sender = new RTCRtpSender(...);sender.start(senderParams);
> CommentsBecause sender (i.e. Alice) sent her parameters that contained
> specific SSRC (and possibly receiver ID) information in the her sender
> parameters, the receiver will latch based upon exact SSRC matching.Proposed
> Capabilities Based APIStep 1: (Alice)var senderCaps =
> RTCRtpSender.getCapbilities();mysignal(senderCaps); Step 2: (Bob)var
> senderCaps = signal();var receiverParams =
> RTCRtpReceiver.createParameters("video", senderCaps);var receiver = new
> RTCRtpReceiver(...);receiver.start(receiverParams);
> mysignal(receiverParams.receiverCapabilities); Step 3: (Alice)var track =
> myObtainMediaTrack();var receiverCaps = mysignal();var senderParams =
> RTCRtpSender.createParameters(track, receiverCaps);var sender = new
> RTCRtpSender(...);sender.start(senderParams); CommentsReceiver (Bob) can
> match an incoming stream because the payload types will match and therefore
> the incoming stream will latch to the receiver based on payload type alone.
> Alice wishes to send media to Bob Using Unhandled EventingCurrent Parameter
> Based APIStep 1: (Alice)var track = myObtainMediaTrack();var senderCaps =
> RTCRtpSender.getCapabilities();var senderParams =
> RTCRtpSender.createParameters(track, senderCaps);mysignal(senderParams);
> Step 2: (Bob)var senderParams = mysignal();var receiverCaps =
> RTPRtcReceiver.getCapabilities();var templateReceiverParams =
> RTPRtcReceiver.filterParameters(senderParams,
> receiverCaps);templateReceiverParams.encodings[0].receiverId =
> "";templateReceiverParams.encodings[0].ssrc = null;var listener =
> RTCRtpListener(...);listener.onunhandledrtp = function(event) {  var
> receiver = new RTCRtpReceiver(...);
>  receiver.start(templateReceiverParams);} mysignal(receiverParams); Step 3:
> (Alice)var receiverParams = mysignal();var senderParams =
> RTPRtcSender.filterParameters(receiverParams, senderCaps);var sender = new
> RTCRtpSender(...);sender.start(senderParams); CommentsBecause sender (i.e.
> Alice) sent her parameters that contained specific SSRC (and possibly
> receiver ID) information in the her sender parameters, the receiver must
> override the template receiver params and remove the exact SSRC to attach
> the incoming stream by payload type.Proposed Capabilities Based APIStep 1:
> (Alice)var senderCaps = RTCRtpSender.getCapbilities();mysignal(senderCaps);
> Step 2: (Bob)var senderCaps = signal();var listener =
> RTCRtpListener(...);listener.onunhandledrtp = function(event) {  var
> receiverParams = RTCRtpReceiver.createParameters("video", senderCaps);  var
> receiver = new RTCRtpReceiver(...);  receiver.start(receiverParams);}
> mysignal(receiverParams.receiverCapabilities); Step 3: (Alice)var track =
> myObtainMediaTrack();var receiverCaps = mysignal();var senderParams =
> RTCRtpSender.createParameters(track, receiverCaps);var sender = new
> RTCRtpSender(...);sender.start(senderParams); CommentsReceiver (Bob) can
> match an incoming stream because the payload types will match and therefore
> the incoming stream will latch to the receiver based on payload type
> alone.Alice / Bob simultaneously exchange information in parallelTo avoid
> requiring a sequential offer / answer exchange, Alice and Bob wish to
> simultaneously exchange their RTC information to receiver media from the
> other party.Current Parameter Based APIStep 1: (Alice / Bob)// [Alice]var
> aliceTrack = myObtainMediaTrack();var aliceSenderCaps =
> RTCRtpSender.getCapabilities();var aliceSenderParams =
> RTCRtpSender.createParameters(aliceTrack, aliceSenderCaps);var
> aliceReceiverCaps = RTCRtpReceiver.getCapabilities();var
> aliceReceiverParams = RTCRtpReceiver.createParameters("video",
> aliceReceiverCaps);mysignal(aliceSenderParams);
> mysignal(aliceReceiverParams); // [Bob]var bobTrack =
> myObtainMediaTrack();var bobSenderCaps = RTCRtpSender.getCapabilities();var
> bobSenderParams = RTCRtpSender.createParameters(bobTrack,
> bobSenderCaps);var bobReceiverCaps = RTCRtpReceiver.getCapabilities();var
> bobReceiverParams = RTCRtpReceiver.createParameters("video",
> bobReceiverCaps);mysignal(bobSenderParams); mysignal(bobReceiverParams);
> Step 2: (Alice / Bob)// [Alice]var bobSenderParams = mysignal();var
> bobReceiverParams = mysignal();bobSenderParmas =
> RTCRtpReceiver.filterParams(bobSenderParams,
> aliceReceiverCaps);bobSenderParmas.encodings[0].receiverId =
> "";bobSenderParmas.encodings[0].ssrc = null;bobSenderParams =
> myFixPayloadTypes(bobSenderParmas, aliceReceiverParams);var aliceReceiver =
> new RTCRtpReceiver(...);aliceReceiver.receive(bobSenderParams);
> bobReceiverParams = RTCRtpSender.filterParams(bobReceiverParams,
> aliceSenderCaps);var aliceSender = new
> RTCRtpSender(...);aliceSender.send(bobReceiverParams); // [Bob]var
> aliceSenderParams = mysignal();var aliceReceiverParams =
> mysignal();aliceSenderParmas =
> RTCRtpReceiver.filterParams(aliceSenderParams,
> bobReceiverCaps);aliceSenderParmas.encodings[0].receiverId =
> "";aliceSenderParmas.encodings[0].ssrc = null;aliceSenderParams =
> myFixPayloadTypes(aliceSenderParmas, bobReceiverParams);var bobReceiver =
> new RTCRtpReceiver(...);bobReceiver.receive(aliceSenderParams);
> aliceReceiverParams = RTCRtpSender.filterParams(aliceReceiverParams,
> aliceSenderCaps);var bobSender = new
> RTCRtpSender(...);bobSender.send(aliceReceiverParams);
> //--------------------------------- // [Alice and Bob need this
> method]function myFixPayloadTypes(senderParams, originalReceiverParams) {
>   // TODO: loop through sender params and then secondarily loop through
>   // original receiver params and set the sender payload type based upon
>   // what is found in the receiver params.   // ...   return
> myFixedSenderParams;} CommentsThe sender includes exact SSRC information
> and signals that to the remote receiver. The issue is the actual sender is
> going to base it's sending params upon the receiver params of the remote
> party which do not contain a specific SSRC (or contains a different SSRC).
> Thus the SSRC has to be stripped from the received sender params or they
> will not match and the receiver won't latch onto the incoming stream as the
> latching must occur by payload type instead.The secondary problem is that
> the sender is actually using the payload types as defined by the remote
> party's receiver but the receiver is basing the payload types based upon
> the remote party's sender. This means the payload types might mismatch and
> the latching based on payload types may not occur. To fix this problem the
> web developer has to fix either the sender's payload types or the
> receiver's payload type.Proposed Capabilities Based APIStep 1: (Alice /
> Bob)// [Alice]var aliceSenderCaps = RTCRtpSender.getCapbilities();var
> aliceReceiverCaps =
> RTCRtpReceiver.getCapabilities();mysignal(aliceSenderCaps);
> mysignal(aliceReceiverCaps); // [Bob]var bobSenderCaps =
> RTCRtpSender.getCapbilities();var bobReceiverCaps =
> RTCRtpReceiver.getCapabilities();mysignal(bobSenderCaps);
> mysignal(bobReceiverCaps); Step 2: (Alice / Bob)// [Alice]var bobSenderCaps
> = mysignal();var bobReceiverCaps = mysignal();var aliceTrack =
> myObtainMediaTrack();var aliceReceiverParams =
> RTCRtpReceiver.createParameters("video", bobSenderCaps);var aliceReceiver =
> new RTCRtpReceiver(...);aliceReceiver.receiver(aliceReceiverParams); var
> aliceSenderParams = RTCRtpSender.createParameters(aliceTrack,
> bobReceiverCaps);var aliceSender = new
> RTCRtpSender(...);aliceSender.send(aliceSenderParams); // [Bob]var
> aliceSenderCaps = mysignal();var aliceReceiverCaps = mysignal();var
> bobTrack = myObtainMediaTrack();var bobReceiverParams =
> RTCRtpReceiver.createParameters("video", aliceSenderCaps);var bobReceiver =
> new RTCRtpReceiver(...);bobReceiver.receiver(bobReceiverParams); var
> bobSenderParams = RTCRtpSender.createParameters(bobTrack,
> aliceReceiverCaps);var bobSender = new
> RTCRtpSender(...);bobSender.send(bobSenderParams); CommentsThe receiver is
> able to latch onto the sender based on payload type alone. Unlike the
> current API, there's no need to strip SSRCs and no need to fiddle and fix
> the payload type. The code is cleaner and clearer as to what's going on and
> does not presume the application level programmer has to know why payload
> types need to match or why SSRCs need to be stripped.Alice wants to use a
> SVC (Scalable Video Codec) to send to BobThis is for illustration purposes
> only. Typical benefits of SVC are greater in conference scenarios rather
> than traditional point to point scenarios. However, this scenario can
> presume that an intermedia conferencing bridge would be between Alice and
> Bob.Current Parameter Based APIStep 1: (Alice)var senderCaps =
> RTCRtpSender.getCapabilities();mySignal(senderCaps); Step 2: (Bob)var
> senderCaps = mysignal();var receiverCaps =
> RTPRtcReceiver.getCapabilities();var receiverParams =
> RTPRtcReceiver.createParameters("video", receiverCaps);var receiverParams =
> RTPRtcReceiver.filterParams(senderCaps);var receiverParams =
> mySetupSVC(receiverParams);var receiver = new
> RTCRtpReceiver(...);receiver.start(receiverParams);
> mysignal(receiverParams); function mySetupSVC(receiverParams) {  // 1.
> search the receiver params for a codec capable of SVC based on
> pre-knowledge  //    of the codec types  // 2. setup SVC params based on
> codec's capabilities    // TODO - step 1 - code needs to be added here to
> do this logic  var chosenCodec = "h264svc"; // hard code for now  // TODO:
> Not sure this code is even right. How does this layer scale even work?  //
> How is temporal and spatial layering defined together? Don't see a knob for
>  // setting up temporal SVC…  receiverParams.receiverId = "foo";
>  receiverParams.encodings[0] = {    "codecName": chosenCodec,    "scale":
> 0.125,    "encodingId": "0"  };  receiverParams.encodings[1] = {
>    "scale": 0.25,    "dependencyEncodingIds": {"0"}  };
>  receiverParams.encodings[2] = {    "scale": 0.5,
>    "dependencyEncodingIds": {"0", "1"}  };} Step 3: (Alice)var
> receiverParams = mysignal();var senderParams =
> RTPRtcSender.filterParameters(receiverParams, senderCaps);var track =
> myObtainMediaTrack();var sender = new RTCRtpSender(track,
> ...);sender.start(senderParams); CommentsThe application developer has to
> have a ton of presumed knowledge about available codecs, codec capabilities
> and needs to have a deep understanding of how the engine interprets the
> layering information. The sender cannot setup the SVC parameters desired
> because it doesn't know the receiver capabilities.The sample above may not
> work for SVC codecs which put each layer on a unique SSRC because the
> receiver did not necessarily pre-dictate the expected SSRCs on each layer
> so the application developer would have to handle this situation too and
> assign SSRCs for each layer manually based on knowledge that the codec
> behaves in this manner.The method to setup temporal or quality SVC is
> unclear. Appropriate parameter knobs for the application developer appear
> to be missing.Proposed Capabilities Based APIStep 1: (Alice)var senderCaps
> = RTCRtpSender.getCapbilities();var senderPrefs = {  "receiverId": "foo",
>  "frameRateScalabilityOptions": {"layers": 2},
>  "scalingScalabilityOptions": {"layers": 2},}; mysignal(senderCaps);
> mysignal(senderPrefs); Step 2: (Bob)var senderCaps = signal();var
> senderPrefs = signal();var receiverParams =
> RTCRtpReceiver.createParameters("video", senderCaps, senderPrefs);var
> receiver = new RTCRtpReceiver(...);receiver.start(receiverParams);
> mysignal(receiverParams.receiverCapabilities); Step 3: (Alice)var track =
> myObtainMediaTrack();var receiverCaps = mysignal();var senderParams =
> RTCRtpSender.createParameters(track, receiverCaps, senderPrefs);var sender
> = new RTCRtpSender(track, ...);sender.start(senderParams); CommentsThe
> application developer doesn't require pre-knowledge of the codecs. The
> developer can quickly and easily specify the types of SVC properties
> desired with much simpler knobs. The developer doesn't have to worry if a
> codec is assigning each layer a unique SSRC or not of if the layering ends
> up being dynamic or not.ConclusionOverall the proposed capabilities based
> API has strong advantages. Main advantages are: 1. Simplicity in setup
> based on "preferences" for the application developer2. Less brittle
> designs/implementations since low level parameters are not exchanged,
> filtered, and interpreted by different browser engines3. Much less
> knowledge (and often no pre-knowledge) is required for the application
> developer to take full advantage of a browser's capabilitiesThere's no
> strong reason to maintain the current API. The biggest difference will be
> that browsers will need to generate compatible parameters based on
> capabilities but that also comes at a big advantage of the browser engines
> not needing to interpreting and filtering low level parameters from other
> browser engines. Both new and current use low level parameters to receive
> or send information so that design aspect remains unchanged.Advantages of
> Current Parameter Based API 1. Browser engines do not need to generate
> parameter from capabilities in a "compatible" manner (although low level
> parameters do need to be filtered in a "compatible" manner so this is not a
> strong advantage).Disadvantages of Current Parameter Based API 1.
> Application developer needs pre-knowledge of SVC codecs to be able to chose
> and setup their properties based upon pre-knowledge of codec capabilities2.
> Application developer needs deep understanding of how layering works to
> setup the layering properties correctly3. Browser engines need to agree on
> how to filter low level parameters based upon capabilities in a consistent
> manner across browsers to ensure compatibility4. Browser engines need to
> agree how to interpret low level parameter objects that were generated by
> other browsers (or other applications)5. Low level parameter based
> exchanges introduce greater brittleness between browsers since extending
> the parameters details could mean breaking existing implementations
> (instead of capabilities which are typically ignored when not understood)6.
> Less innovation / greater brittleness for anything that requires parameter
> object extensions since many browsers as well as applications will be
> fiddling, exchanging, and filtering these low level parameter objects.7.
> Simulcasting with layering doesn't appear to be supported or it's not
> obvious how to set up those scenarios.8. Unclear how to mix and match
> different SVC modes (e.g. temporal, spatial, and quality)9. The application
> developer is uncertain based upon their preferences what the browser engine
> is capable of delivering (without deep understanding of all codecs and
> their properties).10. Header extensions will need manual setup by the
> application developer despite not knowing that codecs or the engines might
> need certain extensions to take advantage of codec features or browser
> engine features.Advantages Proposed Capabilities Based API 1. Application
> developer can easily setup SVC without needing detailed understanding2.
> Typical and even advanced use cases do not require a deep understand of RTC
> to be able to take advantages of capabilities3. Less brittle
> implementations as low level parameter objects are only consumed local by
> the browsers that generate them or only in situations where specific
> compatibilities with legacy systems are required which the default
> generated low level properties read would not be compatible.4. Simulcast
> with layering is supported5. Easy for application developer to mix and
> match different SVC modes (e.g. temporal, spatial, and quality)6. Easy to
> extend support for alternative SVC scalability modes (e.g. colour depth,
> sharpness, ROI)7. Application developer knows what the browser engine is
> capable of delivering given a set of preferences (from resultant
> preferences as returned from "createParameters(...)" 8. Header extensions
> can be automatically set up based on needs and capabilities of the
> browser's RTP engines and codecs.Disadvantages Proposed Capabilities Based
> API 1. Browser engines need to agree on how to compute "compatible"
> parameters for a given codec and media preferences. The rules for
> generation of parameters must be clear.Equal Capabilities of Current and
> Proposed Based API 1. Application developer can always tweak low level
> properties on an "as needed" basis for compatibility2. Both new and current
> proposals send and receive based on lower level parameters (this does not
> change).*
>
>
>
>
>