[mediacapture-main] Add constraint to avoid inefficient (compressed) pixel formats (#739)

henbos has just created a new issue for https://github.com/w3c/mediacapture-main:

== Add constraint to avoid inefficient (compressed) pixel formats ==
Capturing in compressed pixel formats such as MJPEG adds CPU overhead because the browser has to convert (decompress) every frame before delivery to the MediaStreamTrack and beyond.

An application that cares about both quality and performance might ask with non-required constraints for Full HD. If the user has a USB 3.0 camera, Full HD might be delivered without any compression overhead. Great! But if the user has a USB 2.0 camera, due to bus limitations, Full HD would (on cameras available for testing) be captured in MJPEG, adding this overhead. The application pays a performance debt, even though it might have been just as happy if it got HD frames at a lower cost.

TL;DR: Should we add a `{video:{avoidCapturingExpensivePixelFormats:true}}` constraint? I'm not married to the name :)

## Motivation in Numbers

Frames are captured in one format, typically NV12 (420v), YUY2 (yuvs) or MJPEG (dmb1) and then converted. Chromium traditionally converts to I420 (y420) as this format is widely supported by encoders, though it is possible to have other destination pixel formats (e.g. NV12 is supported by some encoders which could allow for a zero-conversion pipeline in WebRTC).

I set up thin "capture and convert" demo ([code](https://chromium-review.googlesource.com/c/chromium/src/+/2450194)) and measured the CPU usage (utilization percentage normalized by CPU frequency using Intel Power Gadget and a [script](https://github.com/henbos/power-gadget) to obtain a sense of the "absolute" amount of work performed).

Here is the result of capturing in various formats*, converting to I420 at 30 fps and measuring the CPU and power consumption.

_* Caveat: NV12 and YUY2 are captured with the built-in MacBook Pro camera and MJPEG is captured using an external Logitech Webcam C930e. The external webcam could contribute to some of the added CPU usage and power consumption, so it would be good to compare MJPEG on webcam with YUY2 on the same webcam, but the majority of the work is in the pixel conversions._

Capture Format | Resolution | Normalized CPU Usage [M cycles/s] | Power Consumption [Watt]
-- | -- | -- | --
NV12 (420v) | 640x480 (VGA) | 26.51 | 3.10
... | 1280x720 (HD) | 28.94 | 3.23
YUY2 (yuvs) | 640x480 (VGA) | 20.57 | 2.98
... | 1280x720 (HD) | 30.97 | 3.31
MJPEG (dmb1) | 640x480 (VGA) | 52.85 | 4.85
... | 1280x720 (HD) | 67.99 | 5.28
... | 1920x1080 (Full HD) | 102.41 | 6.27

Note: I am not measuring the entire browser, I am only measuring a demo that does capturing and conversion.

In this example...
- Capturing in YUY2 at HD instead of MJPEG at Full HD reduces the CPU usage from 102.41 to 30.97 M cycles/s (-70%) and the power consumption from 6.27 to 3.31 W (-47%). We save ~3 W.
- Capturing in YUY2 at HD instead of MJPEG at HD reduces the CPU usage from 67.99 to 30.97 M cycles/s (-54%) and the power consumption from 5.28 to 3.31 W (-37%). We save ~2 W.

## Proposal

Add a new proposal, e.g. BooleanConstraint avoidCapturingExpensivePixelFormats, that if true allows the browser to skip pixel formats of a device that are deemed inefficient (e.g. MJPEG) if that same device supports capturing in other pixel formats.

On Logitech Webcam C930e, where Full HD is only available as MJPEG but 1024x576 and below is available as YUY2, getUserMedia would pick a lower resolution but avoid MJPEG.

P.S. This could result in a tradeoff between frame rate and resolution, more discussion needed.

Please view or discuss this issue at https://github.com/w3c/mediacapture-main/issues/739 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 21 October 2020 15:44:24 UTC