- From: Harald Alvestrand via GitHub <sysbot+gh@w3.org>
- Date: Mon, 16 Oct 2023 10:40:08 +0000
- To: public-webrtc-logs@w3.org
> > Solving the SDP negotiation problem for built-in Sframe is not sufficient. We have to solve it for script transforms. > > That is probably where we have a disconnect. Script transforms have several potential use cases: > > * Implement SFrame or a variant of SFrame. Once we have the packetization format in the UA, I do not see a need for a new generic mechanism to negotiate it. I am ok with SFrame packetization format spec to expose an extension point in the SDP, and we would expose this in our API (say `setCodecPreferences` for instance). Specific interfaces that can only be used for the SFrame transform should go into the SFrame transform specification, not here. setCodecPreferences chooses the preference by the receiver about what the sender should send, it does not enforce either what the sender sends or what the receiver will have to handle. setDepacketizer(PT, depacketizer) is an API that addresses the use case on the receiver side. > * Add metadata to media content. In this case, I believe we want encoded content to stick to the same format (e.g. H264 would stay H264 and metadata would be put in SEI). We do not need to change the packetization format and I do not see the need to negotiate anything in the SDP. RTP header extensions might be a more meaningful extension point if need be. We don't agree here. There is no generic metadata mechanism, and a number of codecs lack functionality for adding metadata. > * Plug a new codec. It seems best to solve this with a separate API, I do not think we want to enshrine this kind of support in encoded transforms. I don't agree here. I think we want to allow for experimentation of this kind, we should actively ensure that we make it possible to try out new codecs without changing the browser. > This `plug a new codec` API would probably need to let the web application handle both the encoding/decoding (UA does not know the content format) and the associated packetization (UA does not know the format so does not know the associated packetization format), in addition to SDP negotiation. Relying on the SFrame packetization for new codecs is probably not something we want, as per IETF feedback. > > For this `plug a new codec` work, I would tend to start with a `plug your own encoder` API that could be used without packetization/SDP handling, for instance to let web apps fine tune a WebCodecs encoder setup (set QP per frame/macro-block for instance). Doing this without SDP handling would make the current situation, where people send content that does not conform to the SDP describing what they claim to be sending, strictly worse. I would strongly oppose adding such an API to the PeerConnection family of APIs without solving the SDP issue first. > We could then address new formats on top of this API to handle SDP negotiation/RTP packetization. It could also be used for the `metadata to media content` use case since web applications may often carry VideoFrame and associated metadata together. If we have to handle SDP negotiation and RTP packetization for this use case, and we know that we have other use cases we need to handle it for, why not accept the current proposal into the WD? -- GitHub Notification of comment by alvestrand Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/pull/186#issuecomment-1764194781 using your GitHub account -- Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config
Received on Monday, 16 October 2023 10:40:10 UTC