[mediacapture-main] Consider discussing content hints for MediaStreamTrack in next interrim

pbos has just created a new issue for https://github.com/w3c/mediacapture-main:

== Consider discussing content hints for MediaStreamTrack in next interrim ==
In WebRTC in Chromium, content is treated differently whether it's screencast or usb webcam devices. It's assumed that screencast can't be downscaled, or text will lose its intelligibility, while usb video class devices webcam content that can be downscaled to preserve frame rate under low bitrate conditions.

This assumption doesn't hold when the screencast content is of a video game, movie or other high-motion content. It doesn't either hold when the usb video device is a capture card, and the capture card is streaming a text website (downscaling this content like normal video will lose intelligibility).

To override this, I created an experimental feature (http://crbug.com/653531, https://wicg.github.io/mst-content-hint/) that permits overriding this assumption (`MediaStreamTrack.contentHint = 'motion';` is treated as "webcam video"). There's also a demo that shows the difference between detail and motion video at https://webrtc.github.io/samples/src/content/capture/video-contenthint/ (requires enabling experimental web features in Chrome). In our suggestion, this was put on the MediaStreamTrack (as opposed to RTCRTPSender) since that has the widest potential use (a MediaStreamRecorder implementation could read it, without having to rewrite the the MediaStreamRecorder standard). It also happened to be the easiest place for us to implement it in Chromium at the time.

There are also values for audio MSTs in the draft doc (music / voice) that could be used for guiding noise reduction algorithms (current noise reduction algorithms will not work well on drums etc. which looks like noise, so it might be best to turn off for music content).

I believe the next step is to hand this suggestion over to this working group (including the question of whether the property fits on MediaStreamTrack as opposed to RTCRtpSender etc.). This doesn't really fit well as a stand-alone document. Can this be discussed in the next interrim meeting? I'd like to have a plausible roadmap out of this feature being experimental, and I believe it's crucial for properly taking care of content at low bitrates (when `screencast == detail` or `usb == motion` fails as a heuristic).

Please view or discuss this issue at https://github.com/w3c/mediacapture-main/issues/478 using your GitHub account

Received on Thursday, 24 August 2017 00:02:12 UTC