RE: Screen sharing and device pixel ratio

In this case this isn't an aspect ratio issue, the pixels are still square, they just have higher density.
But I agree it fits the same reasoning, the transport shouldn't know about transforms necessary on the pixels, it should just deal with "simple" pixels.

Now for the density, as an application I might still like to access the non-scaled stream locally, so I'd like a knob saying give me physical pixels, or give me logical pixels.
Maybe logical pixels should be the default? This is a breaking change with existing implementations.

Mathieu

-----Original Message-----
From: Cullen Jennings (fluffy) [mailto:fluffy@cisco.com] 
Sent: Thursday, December 10, 2015 15:26
To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
Cc: public-media-capture@w3.org
Subject: Re: Screen sharing and device pixel ratio


I think we decided for WebRTC we would only do support square pixels over the wire.  I wonder if we should make the convention for screen capture that if the pixels are not square on the screen, the browser will scale them to square before putting them in the stream.  

This seems somewhat consistent with when the browser display an image on the non square pixels, it is going to scale the image so it is still square on the screen. 

This would all the JS application to assume square pixels. Given most application are going to assume the pixels are square and not consider other things, this would probably be a good default even if we decide to support non square pixels. 


> On Dec 9, 2015, at 9:40 PM, Mathieu Hofman <Mathieu.Hofman@citrix.com> wrote:
> 
> Hi,
> 
> The screen sharing spec doesn't mention anything about how to handle sources with various device pixel ratios.
> 
> I believe current implementations simply capture the physical pixels without taking into consideration the physical to virtual pixel ratio of the source.
> 
> E.g. if a full screen source of 1440 x 900 has a ratio of 2 dppx (Mac retina display), the captured stream will have a size of 2880x1800.
> 
> As an application I might not want a size that large, especially when transmitting to another display with a different pixel ratio (zoom 200%).
> 
> Existing max width / height constraints are not adapted to dealing with the resolution in this case:
> - The content is really meant to be scaled by 2, not a floating point number that would make it just fit in the constrained size
> - A small application window might have a size below the max constraints
> 
> One workaround with the state of the current specs could be to:
> - gDM
> - Read width / height of stream
> - use Window.devicePixelRatio API [1] to guess aspect ratio of content
> - applyConstraints on stream to scale it by that ratio
> 
> Unfortunately this might fail on multi-monitor systems that have different pixel ratios (Windows 10), when the browser tab and content captured are on different monitors. Especially because windows can be dragged across monitors, dynamically changing pixel ratios!
> 
> What I would like is a sort of "constraint" on the gDM call to indicate to the browser I need a stream "scaled" to be in virtual pixels instead of physical pixels.
> 
> Additionally, I would also consider adding a way to "read" the current pixel ratio of the source. This is to know if the captured stream is currently scaled when I ask for a "virtual pixels" stream, or the factor by which to scale it when asking the "physical pixels" and I want to do that processing on my own (e.g. I want to compute a scaling factor adapted to the pixel ratio of the receiving display)
> 
> Would there be any issues adding these to the spec?
> 
> Mathieu
> 
> [1] https://drafts.csswg.org/cssom-view/#dom-window-devicepixelratio
> 
>

Received on Friday, 11 December 2015 02:15:16 UTC