[webrtc-insertable-streams] How to handle transforms largely changing frame size (#50)

youennf has just created a new issue for https://github.com/w3c/webrtc-insertable-streams:

== How to handle transforms largely changing frame size ==
Transforms may be able to introduce large changes in frame size (decrease or increase).
It seems interesting to understand how to handle these cases.

I can see different variations of these cases:

1. Metadata size is not really negotiable by the JS transform, size change is more or less fixed.
The user agent can handle it.
User Agent can detect the size of the metadata and update the encoder bitrate according to the average of the transformed data.

2. Transform may decided to decrease metadata size
The transform may add more or less metadata based on available bandwidth.
The transform could be made aware of the target bit rate from network side, maybe the encoder target bit rate as well.
The transform would then compute how much space it can add to the frame.

3. Transform might want to trade media quality to include more metadata
In that case, the transform can decide whether to reduce the encoder bit rate or the size of the metadata, or both.
It seems useful to notify the JS whenever change of the encoder bit rate is planned and potentially allow the JS to override the default behavior.

Case 1 requires no new API.
Case 2 could be implemented as getter APIs.
Case 3 can be implemented in various ways: ReadableStream/WritableStream pair, transform, events, maybe even through frame dedicated fields). It seems all these variants should provide roughly the same functionality support.

I feel like a single object that the JS could use for all of this when processing a frame might be the more convenient from a web developer perspective.

Please view or discuss this issue at https://github.com/w3c/webrtc-insertable-streams/issues/50 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 7 December 2020 16:11:25 UTC