Fwd: Experimenting with video processing pipelines on the web

I thought Francois' write up on combining various of the media & 
real-time API for video processing would be of interest to this group 
too (see below)

Dom


-------- Message transféré --------
Sujet : Re: Experimenting with video processing pipelines on the web
Date : Tue, 28 Mar 2023 17:03:40 +0000
De : Francois Daoust <fd@w3.org>
Répondre à : Francois Daoust <fd@w3.org>
Pour : public-media-wg@w3.org <public-media-wg@w3.org>
Copie à : Dominique Hazaël-Massieux <dom@w3.org>, Bernard Aboba 
<Bernard.Aboba@microsoft.com>, Peter Thatcher <pthatcher@microsoft.com>

Hi all,

Following my sharing this experimentation with the Media Working Group 
back in December, Chad Hart and Philipp Hancke invited us to develop our 
explorations in a webrtcHacks article. We ended up writing an article in 
two parts, the latter of which got published today:

Part 1 - Real-Time Video Processing with WebCodecs and Streams: 
Processing Pipelines
https://webrtchacks.com/real-time-video-processing-with-webcodecs-and-streams-processing-pipelines-part-1/

Part 2 - Video Frame Processing on the Web – WebAssembly, WebGPU, WebGL, 
WebCodecs, WebNN, and WebTransport
https://webrtchacks.com/video-frame-processing-on-the-web-webassembly-webgpu-webgl-webcodecs-webnn-and-webtransport/

I'm taking the liberty to shamelessly plug these articles here as 
writing them forced us to dig more into details and pushed us to play 
with additional technologies along the way, and I thought some of it 
might be of interest to the group, or at least relevant to discussions 
on possible evolutions of media pipeline architectures. On top of 
learning more about WebGPU (and eventually understanding that there is 
no need to wait before creating a new VideoFrame out of the processed 
one as I thought initially), the demo now also features basic processing 
with pure JavaScript code, with WebAssembly, and a couple of identity 
transforms meant to force CPU-to-GPU and GPU-to-CPU copies, to help 
measure and reflect on performance and memory copies. The second article 
summarizes some of our takeaways.

Thanks,
Francois.



------ Original message ------
From: "Francois Daoust" <fd@w3.org>
To: "Bernard Aboba" <bernard.aboba@microsoft.com>; "Peter Thatcher" 
<pthatcher@microsoft.com>; "public-media-wg@w3.org" <public-media-wg@w3.org>
Cc: "Dominique Hazaël-Massieux" <dom@w3.org>
Date: 20/12/2022 12:10:03

>Hi Bernard, Peter ,
>Hi Media WG,
>
>A couple of months ago, prompted by discussions in this group on the evolutions of the media pipeline architecture and sample code and issues that you shared with the group, Dom and I thought we'd get more hands-on as well to better understand how different web technologies can be mixed to create video processing guidelines. Resulting code is at:
>https://github.com/tidoust/media-tests/
>
>... with demo (currently requires Chrome with WebGPU enabled) at:
>https://tidoust.github.io/media-tests/
>
>Following last Media Working Group call, I have now detailed our approachand the issues we bumped into in the README. I also created a couple of issues on the w3c/media-pipeline-arch repository:
>https://github.com/w3c/media-pipeline-arch/issues/7
>https://github.com/w3c/media-pipeline-arch/issues/8
>
>Our exploration is more a neophyte attempt at processing media than an expert take on technologies, and resulting considerations sit at a higher level than the issues you described during last Media working group call. We may also have missed obvious things that would make these considerations moot :)
>
>It's probably not worth taking working group meeting time for this but, if you think that's useful and if people are interested, we could perhaps organize some sort of TPAC-except-it's-not-TPAC informal breakout session to discuss this early 2023, opening up the session to people from other groups.
>
>Thanks,
>Francois.
>

Received on Wednesday, 29 March 2023 05:04:06 UTC