RE: Screen sharing and device pixel ratio

I’m looking for 2 things right now:

-          A getDisplayMedia “constraint” indicating that I would like the video stream sized as virtual/logical pixels instead of physical pixels. The browser would then internally scale the source content by that ratio.

-          Optionally a stream video track attribute, or maybe a stat for the stream, where I could “read” the current devicePixelRatio for the stream source, irrespective of whether I have a stream in physical or virtual pixels (the app should know what it asked for)

Instead we could maybe extend some of the Web Platform to support video streams with pixelRatios != 1:

-          Video MediaStreamTrack would have an internal devicePixelRatio

-          HTMLVideoElement would have a videoDevicePixelRatio attribute allowing you to read the video’s device pixel ratio (similar to videoWidth/Height)

-          PeerConnection would somehow transmit the devicePixelRatio or use that information for its internal down sampling

That seems like a much more complicated approach which I’m not sure of the implications.

Mathieu

From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
Sent: Thursday, December 10, 2015 13:07
To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
Cc: public-media-capture@w3.org
Subject: Re: Screen sharing and device pixel ratio

Okay, so IIUC now, you'd like to know the size, pixel ratio, and perhaps the "zoom setting" (affecting stylesheet) of the originally rendered content to understand how it was scaled to the video resolution, so that you can then post-process for better display downstream.  But, as you mentioned, the source content can change size (e.g., user resizes their window) during a session.  So, this would be per-video frame metadata.

I'm not sure what to propose here.  Perhaps others can comment?

On Wed, Dec 9, 2015 at 9:40 PM, Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>> wrote:
Resolution constraints don’t help at all. I obviously failed to explain this clearly in my original email.

There is no way to actually know the size, virtual or physical, of the content to capture:

-          It can be an application’s window, or the whole monitor, and with proposed selection constraints changes, we wouldn’t even know or control what type of source the user chose to share

-          Even if we constrained to screen only somehow, the user can chose another monitor than the monitor the browser tab is on, rendering window.screenWidth / screenHeight useless.

-          The source can be on a monitor of a different pixel ratio. Windows 10 has independent ratios per monitor.

Since we can’t know the size (or it can change dynamically), resolution constraints, fixed or otherwise are useless.

Is the problem more clear?

From: miu@google.com<mailto:miu@google.com> [mailto:miu@google.com<mailto:miu@google.com>] On Behalf Of Yuri Wiitala
Sent: Wednesday, December 9, 2015 21:37
To: Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>>
Cc: public-media-capture@w3.org<mailto:public-media-capture@w3.org>
Subject: Re: Screen sharing and device pixel ratio

I just realized I misread the spec.  A ConstrainLongRange extends a LongRange, so you can specify min, max, ideal, and exact for width and height.

On Wed, Dec 9, 2015 at 9:27 PM, Yuri Wiitala <miu@chromium.org<mailto:miu@chromium.org>> wrote:
Currently, a fixed resolution can be requested in MediaTrackConstraints.  Also, window.screenWidth and window.screenHeight should return the screen size in device-independent coordinates (i.e., always 1440x900 in your Mac Retina example).  Would this accomplish what you need?

On Wed, Dec 9, 2015 at 8:40 PM, Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>> wrote:
Hi,

The screen sharing spec doesn't mention anything about how to handle sources with various device pixel ratios.

I believe current implementations simply capture the physical pixels without taking into consideration the physical to virtual pixel ratio of the source.

E.g. if a full screen source of 1440 x 900 has a ratio of 2 dppx (Mac retina display), the captured stream will have a size of 2880x1800.

As an application I might not want a size that large, especially when transmitting to another display with a different pixel ratio (zoom 200%).

Existing max width / height constraints are not adapted to dealing with the resolution in this case:
- The content is really meant to be scaled by 2, not a floating point number that would make it just fit in the constrained size
- A small application window might have a size below the max constraints

One workaround with the state of the current specs could be to:
- gDM
- Read width / height of stream
- use Window.devicePixelRatio API [1] to guess aspect ratio of content
- applyConstraints on stream to scale it by that ratio

Unfortunately this might fail on multi-monitor systems that have different pixel ratios (Windows 10), when the browser tab and content captured are on different monitors. Especially because windows can be dragged across monitors, dynamically changing pixel ratios!

What I would like is a sort of "constraint" on the gDM call to indicate to the browser I need a stream "scaled" to be in virtual pixels instead of physical pixels.

Additionally, I would also consider adding a way to "read" the current pixel ratio of the source. This is to know if the captured stream is currently scaled when I ask for a "virtual pixels" stream, or the factor by which to scale it when asking the "physical pixels" and I want to do that processing on my own (e.g. I want to compute a scaling factor adapted to the pixel ratio of the receiving display)

Would there be any issues adding these to the spec?

Mathieu

[1] https://drafts.csswg.org/cssom-view/#dom-window-devicepixelratio