W3C home > Mailing lists > Public > public-webrtc-logs@w3.org > February 2023

Re: [mediacapture-transform] How to handle varying pixel formats (#83)

From: François Daoust via GitHub <sysbot+gh@w3.org>
Date: Thu, 02 Feb 2023 10:24:48 +0000
To: public-webrtc-logs@w3.org
Message-ID: <issue_comment.created-1413503429-1675333487-sysbot+gh@w3.org>
Regarding conversion by the web app, a relatively easy and efficient way of converting to RGBA is through WebGPU (well, "relatively easy" provided you're familiar with a few WebGPU concepts, and "efficient" when the underlying data of the `VideoFrame` is on the GPU). The [external texture sampler](https://www.w3.org/TR/webgpu/#external-texture-sampling) returns pixels in RGBA (or BGRA) with the specified color space, regardless of the pixel format of the external texture.

Here is a code example of [a transformer function that converts a `VideoFrame` to RGBA with WebGPU](https://github.com/tidoust/media-tests/blob/main/ToRGBXVideoFrameConverter.js). Most lines are "boilerplate" code to use WebGPU. The actual conversion is done by the [fragment shader](https://github.com/tidoust/media-tests/blob/f9033d8ffccbce0025ffcee6186b2f12459b8c42/ToRGBXVideoFrameConverter.js#L75-L77) and does not require specific knowledge of pixel formats and conversion formulas:

```
fn frag_main(@location(0) uv : vec2<f32>) -> @location(0) vec4<f32> {
  return textureSampleBaseClampToEdge(myTexture, mySampler, uv);
}
```

-- 
GitHub Notification of comment by tidoust
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/83#issuecomment-1413503429 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config
Received on Thursday, 2 February 2023 10:24:50 UTC

This archive was generated by hypermail 2.4.0 : Saturday, 6 May 2023 21:20:00 UTC