Re: [mediacapture-transform] How to handle varying pixel formats (#83)

Regarding conversion by the web app, a relatively easy and efficient way of converting to RGBA is through WebGPU (well, "relatively easy" provided you're familiar with a few WebGPU concepts, and "efficient" when the underlying data of the `VideoFrame` is on the GPU). The [external texture sampler](https://www.w3.org/TR/webgpu/#external-texture-sampling) returns pixels in RGBA (or BGRA) with the specified color space, regardless of the pixel format of the external texture.

Here is a code example of [a transformer function that converts a `VideoFrame` to RGBA with WebGPU](https://github.com/tidoust/media-tests/blob/main/ToRGBXVideoFrameConverter.js). Most lines are "boilerplate" code to use WebGPU. The actual conversion is done by the [fragment shader](https://github.com/tidoust/media-tests/blob/f9033d8ffccbce0025ffcee6186b2f12459b8c42/ToRGBXVideoFrameConverter.js#L75-L77) and does not require specific knowledge of pixel formats and conversion formulas:

```
fn frag_main(@location(0) uv : vec2<f32>) -> @location(0) vec4<f32> {
  return textureSampleBaseClampToEdge(myTexture, mySampler, uv);
}
```

-- 
GitHub Notification of comment by tidoust
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/83#issuecomment-1413503429 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 2 February 2023 10:24:50 UTC