[webrtc-pc] What is the anatomy of a serialized WebRTC RTCEncodedVideoFrame and RTCEncodedAudioFrame (#2507)

guest271314 has just created a new issue for https://github.com/w3c/webrtc-pc:

== What is the anatomy of a serialized WebRTC RTCEncodedVideoFrame and RTCEncodedAudioFrame ==
Frames should be Serializable #5 https://github.com/alvestrand/webrtc-media-streams/issues/5

introduces `RTCEncodedVideoFrame` and `RTCEncodedAudioFrame` which, in brief, are described as follows

https://github.com/alvestrand/webrtc-media-streams/issues/5#issuecomment-608332969

> Spec wise, it's a frame encoded with whatever codec is being used. In the current Chromium implementation (not yet shipped) we have tested with VP8.
> The concept is to read the data from the readable stream, process it (including analyzing/changing/replacing data), and writing back to the writable stream to put the frame back in the WebRTC stack.

To facilitate 

>  (including analyzing/changing/replacing data)

ran the accompanying tests and logged a frame at `console`
```
RTCEncodedVideoFrame 
{type: "key", 
 timestamp: 0, 
 data: ArrayBuffer(5826), 
 additionalData: ArrayBuffer(5)
}
```

however, the specification does not provide a complete definition and anatomy of an WebRTC encoded frame.

If we do `new TextDecoder().decode(new Uint8Array(data))` what should be the expected output in "human readable" form, given that the WebRTC frame is now serialized?

For a single video frame is the encoded frame SDP; SDP including VP8; SDP including VP8; or only a VP8 encoded frame (where VP8 is the requested or set codec to be used for the `MediaStreamTrack`)?

Kindly provide an overview of precisely what a serialized WebRTC frame consists of, to facilitate 
"analyzing/changing/replacing data" so that am "writing back to the writable stream to put the frame back in the WebRTC stack" correctly, and am able to verify the data is changed, replaced.


Please view or discuss this issue at https://github.com/w3c/webrtc-pc/issues/2507 using your GitHub account

Received on Sunday, 5 April 2020 14:57:22 UTC