[webrtc-encoded-transform] Should we expose packetization level API to RTCRtpScriptTransform? (#131)

youennf has just created a new issue for https://github.com/w3c/webrtc-encoded-transform:

== Should we expose packetization level API to RTCRtpScriptTransform? ==
A few use cases could benefit from exposing some packetization level API to RTCRtpScriptTransform.
https://github.com/w3c/webrtc-encoded-transform/issues/50 could be solved if script would know the max size of an audio packet.
SPacket could be solved if:
- sender transform would know the max size of a video packet and would provide to the write side of the transform information of how to split a given encoded frame.
- receiver transform would know how the frame was concatenated from some packets.

Looking at libwebrtc, it seems that:
- exposing the max size of an audio packet is feasible, though the computation might be racy (MTU might change for instance between the time JS use the size and the time the packet is sent,  might already be an existing issue in today's implementations).
- exposing the max size of the first packet, middle packets and last packet of a video frame is feasible (same issue with being racy)
These sizes would typically be computed from MTU, RTP overhead, RTP headers overhead, as currently done by packetizers.

Getting or setting packet level boundary of an encoded frame could also be done as an attribute in RTCEncodedVideoFrame.

Here is an API example of how this could look like:
```
dictionary MaxVideoPacketSizes {
    required attribute unsigned long maxPayloadSize;
    attribute unsigned long maxFirstPayloadSize;
    attribute unsigned long maxLastPayloadSize;
}

// To compute max packet size
partial interface RTCRtpScriptTransformer {
    Promise<unsigned long> computeMaxAudioPacketSize(RTCEncodedAudioFrame frame);
    Promise<MaxVideoPacketSizes > computeMaxVideoPacketSizes(RTCEncodedVideoFrame frame);
}

// To read/write video packet boundaries
partial interface RTCEncodedVideoFrame {
    attribute sequence<unsigned long> payloadSizes;
}
```

@aboba, @alvestrand, is it too simplistic or too libwebrtc-implementation centric? Any idea about the feasibility or the potential race issues?

Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/131 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 2 February 2022 14:00:37 UTC