RE: Screen sharing and device pixel ratio

Resolution constraints don’t help at all. I obviously failed to explain this clearly in my original email.

There is no way to actually know the size, virtual or physical, of the content to capture:

-          It can be an application’s window, or the whole monitor, and with proposed selection constraints changes, we wouldn’t even know or control what type of source the user chose to share

-          Even if we constrained to screen only somehow, the user can chose another monitor than the monitor the browser tab is on, rendering window.screenWidth / screenHeight useless.

-          The source can be on a monitor of a different pixel ratio. Windows 10 has independent ratios per monitor.

Since we can’t know the size (or it can change dynamically), resolution constraints, fixed or otherwise are useless.

Is the problem more clear?

From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
Sent: Wednesday, December 9, 2015 21:37
To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
Cc: public-media-capture@w3.org
Subject: Re: Screen sharing and device pixel ratio

I just realized I misread the spec.  A ConstrainLongRange extends a LongRange, so you can specify min, max, ideal, and exact for width and height.

On Wed, Dec 9, 2015 at 9:27 PM, Yuri Wiitala <miu@chromium.org<mailto:miu@chromium.org>> wrote:
Currently, a fixed resolution can be requested in MediaTrackConstraints.  Also, window.screenWidth and window.screenHeight should return the screen size in device-independent coordinates (i.e., always 1440x900 in your Mac Retina example).  Would this accomplish what you need?

On Wed, Dec 9, 2015 at 8:40 PM, Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>> wrote:
Hi,

The screen sharing spec doesn't mention anything about how to handle sources with various device pixel ratios.

I believe current implementations simply capture the physical pixels without taking into consideration the physical to virtual pixel ratio of the source.

E.g. if a full screen source of 1440 x 900 has a ratio of 2 dppx (Mac retina display), the captured stream will have a size of 2880x1800.

As an application I might not want a size that large, especially when transmitting to another display with a different pixel ratio (zoom 200%).

Existing max width / height constraints are not adapted to dealing with the resolution in this case:
- The content is really meant to be scaled by 2, not a floating point number that would make it just fit in the constrained size
- A small application window might have a size below the max constraints

One workaround with the state of the current specs could be to:
- gDM
- Read width / height of stream
- use Window.devicePixelRatio API [1] to guess aspect ratio of content
- applyConstraints on stream to scale it by that ratio

Unfortunately this might fail on multi-monitor systems that have different pixel ratios (Windows 10), when the browser tab and content captured are on different monitors. Especially because windows can be dragged across monitors, dynamically changing pixel ratios!

What I would like is a sort of "constraint" on the gDM call to indicate to the browser I need a stream "scaled" to be in virtual pixels instead of physical pixels.

Additionally, I would also consider adding a way to "read" the current pixel ratio of the source. This is to know if the captured stream is currently scaled when I ask for a "virtual pixels" stream, or the factor by which to scale it when asking the "physical pixels" and I want to do that processing on my own (e.g. I want to compute a scaling factor adapted to the pixel ratio of the receiving display)

Would there be any issues adding these to the spec?

Mathieu

[1] https://drafts.csswg.org/cssom-view/#dom-window-devicepixelratio