[mediacapture-image] Background Concealment (Blur/Replacement). (#291)

riju has just created a new issue for https://github.com/w3c/mediacapture-image:

== Background Concealment (Blur/Replacement). ==
### Why ?

[Google Meet](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html), [Microsoft Teams](https://www.microsoft.com/en-us/microsoft-teams/background-blur), [Zoom](https://support.zoom.us/hc/en-us/articles/360061468611-Using-blurred-background-) and every video-conferencing application these days has the Background Concealment /  (Blur, Replacement) so that users can minimize distractions and keep the focus on the subject. Most web based apps will use some form of AI inference to implement this feature, like say Jitsi is using Meet’s model and TensorflowLite’s WASM backend [[commit](https://github.com/jitsi/jitsi-meet/commit/946339a52e2691dee31595e51793a3cdeb95f584)].

The popularity on the native side and also usage of NN frameworks to implement on the Web platform warrants a discussion if it makes sense bring this feature to the Web Platform (WebRTC) in a shape which might benefit all without bringing their own frameworks and leveraging the underlying platform support, which in many cases might be accelerated via VPUs or other ASIC processors.

### How ?

MediaFoundation has added the support for background segmentation using properties like [`KSCAMERA_EXTENDEDPROP_BACKGROUNDSEGMENTATION_BLUR`](https://docs.microsoft.com/en-us/windows-hardware/drivers/stream/ksproperty-cameracontrol-extended-backgroundsegmentation) from Windows 11, if there is support from the underlying driver. By encapsulating the _preferably (ASIC) hardware accelerated inference_ work in the driver and leveraging standard platform APIs, we do not have to re-invent the wheel for every web application.

Apple's [Segmentation Matte](https://developer.apple.com/documentation/avfoundation/avsemanticsegmentationmatte) in Portrait Mode captures are essentially a general framework to implement many new features, one of which can be Background Replacement.

**_Opens_**

Do we try to compose a single API for Background Blur (BB) and Background Replacement (BR) ?




Please view or discuss this issue at https://github.com/w3c/mediacapture-image/issues/291 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 13 October 2021 22:35:31 UTC