RE: Screen sharing and device pixel ratio

Opened issue https://github.com/w3c/mediacapture-screen-share/issues/35


I have some question there on naming of the constraint and how to expose the current device pixel ratio of the captured content.

Mathieu

-----Original Message-----
From: Martin Thomson [mailto:martin.thomson@gmail.com] 
Sent: Thursday, December 10, 2015 13:43
To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
Cc: Yuri Wiitala <miu@chromium.org>; public-media-capture@w3.org
Subject: Re: Screen sharing and device pixel ratio

Mathieu, this seems like a reasonable request.  Can you open an issue on the screensharing spec?  Ideally, also a pull request; I'm unlikely to get to this soon.

(Yuri, please make sure that you have properly joined the mailing list, we're not seeing your mail.)

On 11 December 2015 at 08:28, Mathieu Hofman <Mathieu.Hofman@citrix.com> wrote:
> I’m looking for 2 things right now:
>
> -          A getDisplayMedia “constraint” indicating that I would like the
> video stream sized as virtual/logical pixels instead of physical 
> pixels. The browser would then internally scale the source content by that ratio.
>
> -          Optionally a stream video track attribute, or maybe a stat for
> the stream, where I could “read” the current devicePixelRatio for the 
> stream source, irrespective of whether I have a stream in physical or 
> virtual pixels (the app should know what it asked for)
>
>
>
> Instead we could maybe extend some of the Web Platform to support 
> video streams with pixelRatios != 1:
>
> -          Video MediaStreamTrack would have an internal devicePixelRatio
>
> -          HTMLVideoElement would have a videoDevicePixelRatio attribute
> allowing you to read the video’s device pixel ratio (similar to
> videoWidth/Height)
>
> -          PeerConnection would somehow transmit the devicePixelRatio or use
> that information for its internal down sampling
>
>
>
> That seems like a much more complicated approach which I’m not sure of 
> the implications.
>
>
>
> Mathieu
>
>
>
> From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
> Sent: Thursday, December 10, 2015 13:07
> To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
> Cc: public-media-capture@w3.org
> Subject: Re: Screen sharing and device pixel ratio
>
>
>
> Okay, so IIUC now, you'd like to know the size, pixel ratio, and 
> perhaps the "zoom setting" (affecting stylesheet) of the originally 
> rendered content to understand how it was scaled to the video 
> resolution, so that you can then post-process for better display 
> downstream.  But, as you mentioned, the source content can change size 
> (e.g., user resizes their window) during a session.  So, this would be per-video frame metadata.
>
>
>
> I'm not sure what to propose here.  Perhaps others can comment?
>
>
>
> On Wed, Dec 9, 2015 at 9:40 PM, Mathieu Hofman 
> <Mathieu.Hofman@citrix.com>
> wrote:
>
> Resolution constraints don’t help at all. I obviously failed to 
> explain this clearly in my original email.
>
>
>
> There is no way to actually know the size, virtual or physical, of the 
> content to capture:
>
> -          It can be an application’s window, or the whole monitor, and with
> proposed selection constraints changes, we wouldn’t even know or 
> control what type of source the user chose to share
>
> -          Even if we constrained to screen only somehow, the user can chose
> another monitor than the monitor the browser tab is on, rendering 
> window.screenWidth / screenHeight useless.
>
> -          The source can be on a monitor of a different pixel ratio.
> Windows 10 has independent ratios per monitor.
>
>
>
> Since we can’t know the size (or it can change dynamically), 
> resolution constraints, fixed or otherwise are useless.
>
>
>
> Is the problem more clear?
>
>
>
> From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
> Sent: Wednesday, December 9, 2015 21:37
> To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
> Cc: public-media-capture@w3.org
> Subject: Re: Screen sharing and device pixel ratio
>
>
>
> I just realized I misread the spec.  A ConstrainLongRange extends a 
> LongRange, so you can specify min, max, ideal, and exact for width and 
> height.
>
>
>
> On Wed, Dec 9, 2015 at 9:27 PM, Yuri Wiitala <miu@chromium.org> wrote:
>
> Currently, a fixed resolution can be requested in MediaTrackConstraints.
> Also, window.screenWidth and window.screenHeight should return the 
> screen size in device-independent coordinates (i.e., always 1440x900 
> in your Mac Retina example).  Would this accomplish what you need?
>
>
>
> On Wed, Dec 9, 2015 at 8:40 PM, Mathieu Hofman 
> <Mathieu.Hofman@citrix.com>
> wrote:
>
> Hi,
>
> The screen sharing spec doesn't mention anything about how to handle 
> sources with various device pixel ratios.
>
> I believe current implementations simply capture the physical pixels 
> without taking into consideration the physical to virtual pixel ratio of the source.
>
> E.g. if a full screen source of 1440 x 900 has a ratio of 2 dppx (Mac 
> retina display), the captured stream will have a size of 2880x1800.
>
> As an application I might not want a size that large, especially when 
> transmitting to another display with a different pixel ratio (zoom 200%).
>
> Existing max width / height constraints are not adapted to dealing 
> with the resolution in this case:
> - The content is really meant to be scaled by 2, not a floating point 
> number that would make it just fit in the constrained size
> - A small application window might have a size below the max 
> constraints
>
> One workaround with the state of the current specs could be to:
> - gDM
> - Read width / height of stream
> - use Window.devicePixelRatio API [1] to guess aspect ratio of content
> - applyConstraints on stream to scale it by that ratio
>
> Unfortunately this might fail on multi-monitor systems that have 
> different pixel ratios (Windows 10), when the browser tab and content 
> captured are on different monitors. Especially because windows can be 
> dragged across monitors, dynamically changing pixel ratios!
>
> What I would like is a sort of "constraint" on the gDM call to 
> indicate to the browser I need a stream "scaled" to be in virtual 
> pixels instead of physical pixels.
>
> Additionally, I would also consider adding a way to "read" the current 
> pixel ratio of the source. This is to know if the captured stream is 
> currently scaled when I ask for a "virtual pixels" stream, or the 
> factor by which to scale it when asking the "physical pixels" and I 
> want to do that processing on my own (e.g. I want to compute a scaling 
> factor adapted to the pixel ratio of the receiving display)
>
> Would there be any issues adding these to the spec?
>
> Mathieu
>
> [1] https://drafts.csswg.org/cssom-view/#dom-window-devicepixelratio

>
>
>
>
>
>

Received on Friday, 11 December 2015 23:26:44 UTC