Re: [webrtc-encoded-transform] Encoded frame IDLs share definitions with WebCodecs (#99)

> In general, I would prefer we use move/neuter behavior here, instead of relying on garbage collection.
> This is true for encoded chunks as well as raw data.

For raw frames, we definitely do not want anyone to rely on garbage collection. Both AudioFrame and (more importantly) VideoFrame offer a close() method (similar to ImageBitmap) that releases the object's reference to underlying resources (and may immediately free them if that was the only reference). The spec encourages users to close() immediately when they no longer need the frame (and Chrome actually warns users if it GC's a frame that wasn't closed()). I'm adding further clarity to the ref-counting behavior in https://github.com/w3c/webcodecs/pull/167.

For encoded chunks, GC has been working fine since the data is relatively small, but I'd be open to adding a close() there as well. 

> There might be cases where we could also potentially reuse a buffer (say decryption).

Sorry, I didn't quite follow this. Can you say more?

> The typical flow is: I read a frame F from encoder, I change the frame F, I write the frame F in the packetizer.
> After writing the frame F, the slimmer frame F is, the better.
> Similar to transferring the frame.

With the Encoded*Chunk readInto() PR, this entails one copy when reading from the encoder output. I think it strikes a good balance between keeping I/O types immutable (making it easy to reason about TOCTOU issues) and not being too onerous.

Additionally, for VideoFrame, the pixel data will often reside in GPU memory. We don't want to copy to CPU for the sake of making an ArrayBuffer unless the user explicitly requests that. We achieve this by again offering a readInto() API (this time, making it async to avoid requiring synchronization that hurts GPU pipeline perf). For folks that just want to go from MediaStreamTrack -> Canvas or from VideoDecoder -> Canvas, this API would never be called and everything stays on the GPU.   This lazy async readback option isn't feasible with an interface that just offers an ArrayBuffer.

In the long run, we hope the web platform will introduce a type of read-only ArrayBuffer, which enables us to do things like BYOB and expose buffer attributes (except for the GPU VideoFrame case mentioned above) without fear of TOCTOU issues. Extensive discussion on that in https://github.com/WICG/reducing-memory-copies/issues/1

-- 
GitHub Notification of comment by chcunningham
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/99#issuecomment-824494217 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 22 April 2021 02:33:19 UTC