Re: Experimenting with video processing pipelines on the web

These are great articles. Thanks for writing and sharing!

`copyTo` performance should get better over time in the GPU resource -> GPU
encoder case. I agree it remains hard to reason about now though. The
design goal was always to minimize copies as much as possible when using
the built-in drawing primitives like drawImage, texImage, and
importExternalTexture -- this works better on some GPUs and OSs than
others. E.g., In Chrome, macOS and ChromeOS have comprehensive GPU memory
buffer support available to both the renderer sandbox and gpu process,
making copies and transfers more efficient than other platforms.

Are there any changes you'd propose to WebCodecs based on your experiences?

- dale

On Tue, Mar 28, 2023 at 10:03 AM Francois Daoust <fd@w3.org> wrote:

> Hi all,
>
> Following my sharing this experimentation with the Media Working Group
> back in December, Chad Hart and Philipp Hancke invited us to develop our
> explorations in a webrtcHacks article. We ended up writing an article in
> two parts, the latter of which got published today:
>
> Part 1 - Real-Time Video Processing with WebCodecs and Streams:
> Processing Pipelines
>
> https://webrtchacks.com/real-time-video-processing-with-webcodecs-and-streams-processing-pipelines-part-1/
>
> Part 2 - Video Frame Processing on the Web – WebAssembly, WebGPU, WebGL,
> WebCodecs, WebNN, and WebTransport
>
> https://webrtchacks.com/video-frame-processing-on-the-web-webassembly-webgpu-webgl-webcodecs-webnn-and-webtransport/
>
> I'm taking the liberty to shamelessly plug these articles here as
> writing them forced us to dig more into details and pushed us to play
> with additional technologies along the way, and I thought some of it
> might be of interest to the group, or at least relevant to discussions
> on possible evolutions of media pipeline architectures. On top of
> learning more about WebGPU (and eventually understanding that there is
> no need to wait before creating a new VideoFrame out of the processed
> one as I thought initially), the demo now also features basic processing
> with pure JavaScript code, with WebAssembly, and a couple of identity
> transforms meant to force CPU-to-GPU and GPU-to-CPU copies, to help
> measure and reflect on performance and memory copies. The second article
> summarizes some of our takeaways.
>
> Thanks,
> Francois.
>
>
>
> ------ Original message ------
> From: "Francois Daoust" <fd@w3.org>
> To: "Bernard Aboba" <bernard.aboba@microsoft.com>; "Peter Thatcher"
> <pthatcher@microsoft.com>; "public-media-wg@w3.org"
> <public-media-wg@w3.org>
> Cc: "Dominique Hazaël-Massieux" <dom@w3.org>
> Date: 20/12/2022 12:10:03
>
> >Hi Bernard, Peter ,
> >Hi Media WG,
> >
> >A couple of months ago, prompted by discussions in this group on the
> evolutions of the media pipeline architecture and sample code and issues
> that you shared with the group, Dom and I thought we'd get more hands-on as
> well to better understand how different web technologies can be mixed to
> create video processing guidelines. Resulting code is at:
> >https://github.com/tidoust/media-tests/
> >
> >... with demo (currently requires Chrome with WebGPU enabled) at:
> >https://tidoust.github.io/media-tests/
> >
> >Following last Media Working Group call, I have now detailed our approach
> and the issues we bumped into in the README. I also created a couple of
> issues on the w3c/media-pipeline-arch repository:
> >https://github.com/w3c/media-pipeline-arch/issues/7
> >https://github.com/w3c/media-pipeline-arch/issues/8
> >
> >Our exploration is more a neophyte attempt at processing media than an
> expert take on technologies, and resulting considerations sit at a higher
> level than the issues you described during last Media working group call.
> We may also have missed obvious things that would make these considerations
> moot :)
> >
> >It's probably not worth taking working group meeting time for this but,
> if you think that's useful and if people are interested, we could perhaps
> organize some sort of TPAC-except-it's-not-TPAC informal breakout session
> to discuss this early 2023, opening up the session to people from other
> groups.
> >
> >Thanks,
> >Francois.
> >
>
>