Re: [webrtc-encoded-transform] Revisit question of whether frames need to be ordered on incoming (#119)

I'd like to add my perspective supporting the provision of ordered frames by default:

1. **Specification Alignment**
According to the [W3C specification](https://www.w3.org/TR/webrtc-encoded-transform/#stream-processing), in the receiver's readEncodedData, _"frames produced by rtcObject's packetizer MUST be enqueued in rtcObject.[[readable]] in the same encoder's output order."_ The depacketizer typically uses RTP packet sequence numbers to reorder packets as needed. This clearly indicates that ordered frames are the expected behavior per the standard.

2. **Practical Use Cases Favor Ordered Frames**
  The primary use cases for encoded-transform on the receiver side include:

    1. SEI data processing (for features like whiteboards, subtitles)
    2. Decryption operations
    3. Integration with WebCodecs

    Many of these scenarios are highly dependent on frame ordering. I struggle to identify practical use cases that genuinely require unordered frames. For the low-latency scenarios mentioned by @englishm-mux, could this perhaps be addressed more effectively through JitterBuffer configuration, rather than exposing the complexity of unordered frames to application code?

3. **Redundancy and Complexity of JS-side Buffering**
Maintaining a separate frame buffer queue in JavaScript introduces:

    1. Code redundancy and increased complexity

    2.  Difficulty for developers in correctly determining when to drop frames or process the next frame

    3. Potential for additional bugs and performance issues

In summary, I believe the API should provide ordered frames by default. If there are specific scenarios that require unordered frames, this should be offered as an optional feature through explicit configuration parameters.

-- 
GitHub Notification of comment by shushushv
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/119#issuecomment-3442296101 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 24 October 2025 10:03:33 UTC