ReplaceTrack failure modes (Re: ReplaceTrack vs RTPSender.track = new)

A number of different issues in this message - I'm changing the subject 
again.

Note that in this particular case, I was only considering the issues 
that can happen within the scope of ReplaceTrack - the browser can fire 
negotiationneeded all it wants, but it's the Javascript's decision when 
(and whether) to actually perform a negotiation.

So the specification for ReplaceTrack has to say what ReplaceTrack does 
in advance of negotiation; there's a separate part where we consider 
what negotiation does after ReplaceTrack is called.


On 04/17/2015 07:39 AM, Jan-Ivar Bruaroey wrote:
> Hi Harald, I had some trouble following the example with my knowledge, 
> and have some questions if you don't mind.
>
> On 4/15/15 3:31 AM, Harald Alvestrand wrote:
>> In mulling over this issue, I've been thinking about failure cases, 
>> and about negotiated constraints.
>>
>> Consider this case:
>>
>> A negotiates with B to send video
>> The agreeed capabilities for B is 320x200 max (it's a doorbell, I 
>> guess).
>
> I'm not sure what "negotiated constraints" means. Are we talking about 
> session (e.g. hardware) abilities in SDP here (I assume so since you 
> mention doorbell)?
>
> If so, then doesn't the 320x200 track need to be added first in order 
> to negotiate video?

The Dallas agreement wrt imageattr was that the SDP sent from B to A 
would contain the line
a=imageattr:<type> recv [w=1:320 h=1:200]
(if I remember the syntax right).

>
>> A sends a track to B with 320x200 format.
>
> If the track somehow wasn't added first, but is added later instead, A 
> would need to renegotiate, right? Say it was an HD track, how would it 
> work/fail today?

All this is out of scope for ReplaceTrack. (I think our current 
assumption is that there will be downsampling done inside PeerConnection 
somewhere).

>
>> A does ReplaceTrack with a HD track (currently also being displayed 
>> in HD somewhere else).
>>
>> Three possibilities:
>>
>> - A downsamples to 320x200 as (conceptuaully) an internal function of 
>> the PeerConnection
>
> My understanding of SDP is limited (assuming we are talking about 
> SDP). What does A know about B's abilities beyond the answer, which is 
> a subset of the offer, which, as I understand it, would only (need to) 
> cover the 320x200 track added?

Note that ReplaceTrack has to work the same whether A was the offerer or 
B was the offerer.

>
> Both PRs say "Things like dimensions and frame rate do not require 
> negotiation." - I relied on other people when writing that, but this 
> may assume things about the other end. 

Yep. I think you were advised badly.
> For example, in ideal cases where the recipient is much like the 
> sender (e.g. a local-loop test), we can replaceTrack between tracks 
> with different dimensions and frameRates, and the recipient handles 
> the change without renegotiation (works in Firefox: 
> http://jsfiddle.net/oyabqd0j )! This is powerful, but a doorbell would 
> probably react poorly to this, so how to detect this ahead of time 
> from the SDP?

that's the difference between operating in a homogenous environment 
(even Firefox-to-Chrome is a relatively homogenous environment) and 
making standards that work for everyone...

>
> Does the doorbell wreck it for everybody?
>
>> - A treats B's restrictions as (new) constraints on the track, and 
>> all other users switch to 320x200
>
> In replaceTrack? No, PeerConnections can't change constraints. 
> Constraints is a gUM concept. To a PeerConnection, a track here is a 
> hose with video coming out of it. It can clamp the hose, but otherwise 
> has little control over what comes out of it. The knobs on the hose 
> (applyConstraints) are for the website to mess with, not 
> PeerConnection, so whatever "A" is in the example, it can't be 
> PeerConnection.

"treats as" is the operative word. Check the illustration in section 5 
of getusermedia (which doesn't illustrate one track with multiple 
destinations, though.) The sink can offer limitations that have to be 
considered together with the constraints explicitly set on the track.

>
> If A, the website, wants to change camera constraints, then that's on 
> it to manage (though how would it learn of B's hardware limitations in 
> this case?)
>
>> - ReplaceTrack fails
>
> You skipped renegotiation, but I suspect our current implementation 
> would neither fail nor renegotiate over this (see two up). I confess 
> not having tested a doorbell.

Yes, see top of message.

>
>> Replay the same case where the source of the new track is a VP8-only 
>> camera, and B says it can only take H.264 (I guess it's a gateway).
>>
>> Three possibilities:
>>
>> - A transcodes the video internally to PeerConnection
>
> What do we do today for renegotiation? The PR notes " Reasons 
> negotiation may be needed include video differing in raw vs. 
> pre-encoded format".

See top of message.

>
>> - A inserts transcoding at the source
>
> Again, can't alter the source AFAIK.

Yes, it can, as long as it remains within the constraints demanded by 
all tracks attached to the source. Again, section 5.

>
>> - ReplaceTrack fails
>
> We probably fire negotiationneeded instead here.

But what happens to the track while we wait for the JS to decide whether 
it wants to negotiate or not?
Perhaps "negotiationneeded" is a failure?

>
>> And of course the weird one:
>>
>> A does ReplaceTrack, trying to replace an audio track with a video 
>> track.
>
> This one is covered in both PRs:
>
>   * "1. If withTrack's kind attribute differs from the kind of
>     RTCRtpSender.track, then reject p with
>     IncompatibleMediaStreamTrackError and abort these steps."
>   * "3. If newTrack's kind attribute differs from the kind of
>     oldTrack, then throw an IncompatibleMediaStreamTrackError
>     exception and abort these steps."
>

I'd missed that - somehow I got the idea that assignments aren't 
supposed to throw.

>
> I would love to understand the example, even if the conclusion of the 
> subject is foregone, as it may have wider implications to what's 
> possible with replaceTrack and what's not.

Yes, no matter what the syntax looks like, we have to know exactly what 
actions are performed.

>
> .: Jan-Ivar :.
>