Re: [webrtc-extensions] Suggest API for generating a Stream rather than a MediaStreamTrack from incoming media (#85)

This is the nice thing about discussing ideas while they are still not fully baked; we get to imagine answers, and can incorporate those answers in the final proposal :-)

Lipsync would be up to the processor (consumer of the Stream). It gets all the timestamps, and does the adjustments it wants to do. The processor will have access to the SDP in the PeerConnection, and can read data from there if needed.

It is also responsible for dealing with changes in what codec is being used - if it gets a frame with PT 96 (VP8) followed by a frame with PT 99 (AV1), it will be responsible for feeding the first one to the VP8 decoder and the second one to the AV1 decoder.

The normal case would be that after initial processing (if any), one would feed the frames to a WebCodec decoder. (at least in our implementation, the codec implementations are the same between WebCodec and WebRTC, but the control surface shape is a bit different.)

In the sketch I proposed, if the app wants to grab a stream for video and not for audio, it would lock the video stream but not lock the audio stream; the audio stream would then go back for normal processing (bullet 3). So yes, it would be possible to decide at stream creation time (which is later than configuration time) whether or not you wanted to receive a certain media stream as a Stream or a MediaStreamTrack.

I haven't worked out anything about renegotiation yet; it is possible that it's enough to give an example where the application listens to PeerConnection signaling state and re-performs initialization steps when the state goes to "stable" (end of a new negotiation round) - we may not need new interfaces here.



-- 
GitHub Notification of comment by alvestrand
Please view or discuss this issue at https://github.com/w3c/webrtc-extensions/issues/85#issuecomment-943088194 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 14 October 2021 07:27:19 UTC