Re: Screen sharing and device pixel ratio

Mathieu, this seems like a reasonable request.  Can you open an issue
on the screensharing spec?  Ideally, also a pull request; I'm unlikely
to get to this soon.

(Yuri, please make sure that you have properly joined the mailing
list, we're not seeing your mail.)

On 11 December 2015 at 08:28, Mathieu Hofman <Mathieu.Hofman@citrix.com> wrote:
> I’m looking for 2 things right now:
>
> -          A getDisplayMedia “constraint” indicating that I would like the
> video stream sized as virtual/logical pixels instead of physical pixels. The
> browser would then internally scale the source content by that ratio.
>
> -          Optionally a stream video track attribute, or maybe a stat for
> the stream, where I could “read” the current devicePixelRatio for the stream
> source, irrespective of whether I have a stream in physical or virtual
> pixels (the app should know what it asked for)
>
>
>
> Instead we could maybe extend some of the Web Platform to support video
> streams with pixelRatios != 1:
>
> -          Video MediaStreamTrack would have an internal devicePixelRatio
>
> -          HTMLVideoElement would have a videoDevicePixelRatio attribute
> allowing you to read the video’s device pixel ratio (similar to
> videoWidth/Height)
>
> -          PeerConnection would somehow transmit the devicePixelRatio or use
> that information for its internal down sampling
>
>
>
> That seems like a much more complicated approach which I’m not sure of the
> implications.
>
>
>
> Mathieu
>
>
>
> From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
> Sent: Thursday, December 10, 2015 13:07
> To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
> Cc: public-media-capture@w3.org
> Subject: Re: Screen sharing and device pixel ratio
>
>
>
> Okay, so IIUC now, you'd like to know the size, pixel ratio, and perhaps the
> "zoom setting" (affecting stylesheet) of the originally rendered content to
> understand how it was scaled to the video resolution, so that you can then
> post-process for better display downstream.  But, as you mentioned, the
> source content can change size (e.g., user resizes their window) during a
> session.  So, this would be per-video frame metadata.
>
>
>
> I'm not sure what to propose here.  Perhaps others can comment?
>
>
>
> On Wed, Dec 9, 2015 at 9:40 PM, Mathieu Hofman <Mathieu.Hofman@citrix.com>
> wrote:
>
> Resolution constraints don’t help at all. I obviously failed to explain this
> clearly in my original email.
>
>
>
> There is no way to actually know the size, virtual or physical, of the
> content to capture:
>
> -          It can be an application’s window, or the whole monitor, and with
> proposed selection constraints changes, we wouldn’t even know or control
> what type of source the user chose to share
>
> -          Even if we constrained to screen only somehow, the user can chose
> another monitor than the monitor the browser tab is on, rendering
> window.screenWidth / screenHeight useless.
>
> -          The source can be on a monitor of a different pixel ratio.
> Windows 10 has independent ratios per monitor.
>
>
>
> Since we can’t know the size (or it can change dynamically), resolution
> constraints, fixed or otherwise are useless.
>
>
>
> Is the problem more clear?
>
>
>
> From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
> Sent: Wednesday, December 9, 2015 21:37
> To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
> Cc: public-media-capture@w3.org
> Subject: Re: Screen sharing and device pixel ratio
>
>
>
> I just realized I misread the spec.  A ConstrainLongRange extends a
> LongRange, so you can specify min, max, ideal, and exact for width and
> height.
>
>
>
> On Wed, Dec 9, 2015 at 9:27 PM, Yuri Wiitala <miu@chromium.org> wrote:
>
> Currently, a fixed resolution can be requested in MediaTrackConstraints.
> Also, window.screenWidth and window.screenHeight should return the screen
> size in device-independent coordinates (i.e., always 1440x900 in your Mac
> Retina example).  Would this accomplish what you need?
>
>
>
> On Wed, Dec 9, 2015 at 8:40 PM, Mathieu Hofman <Mathieu.Hofman@citrix.com>
> wrote:
>
> Hi,
>
> The screen sharing spec doesn't mention anything about how to handle sources
> with various device pixel ratios.
>
> I believe current implementations simply capture the physical pixels without
> taking into consideration the physical to virtual pixel ratio of the source.
>
> E.g. if a full screen source of 1440 x 900 has a ratio of 2 dppx (Mac retina
> display), the captured stream will have a size of 2880x1800.
>
> As an application I might not want a size that large, especially when
> transmitting to another display with a different pixel ratio (zoom 200%).
>
> Existing max width / height constraints are not adapted to dealing with the
> resolution in this case:
> - The content is really meant to be scaled by 2, not a floating point number
> that would make it just fit in the constrained size
> - A small application window might have a size below the max constraints
>
> One workaround with the state of the current specs could be to:
> - gDM
> - Read width / height of stream
> - use Window.devicePixelRatio API [1] to guess aspect ratio of content
> - applyConstraints on stream to scale it by that ratio
>
> Unfortunately this might fail on multi-monitor systems that have different
> pixel ratios (Windows 10), when the browser tab and content captured are on
> different monitors. Especially because windows can be dragged across
> monitors, dynamically changing pixel ratios!
>
> What I would like is a sort of "constraint" on the gDM call to indicate to
> the browser I need a stream "scaled" to be in virtual pixels instead of
> physical pixels.
>
> Additionally, I would also consider adding a way to "read" the current pixel
> ratio of the source. This is to know if the captured stream is currently
> scaled when I ask for a "virtual pixels" stream, or the factor by which to
> scale it when asking the "physical pixels" and I want to do that processing
> on my own (e.g. I want to compute a scaling factor adapted to the pixel
> ratio of the receiving display)
>
> Would there be any issues adding these to the spec?
>
> Mathieu
>
> [1] https://drafts.csswg.org/cssom-view/#dom-window-devicepixelratio
>
>
>
>
>
>

Received on Thursday, 10 December 2015 21:43:43 UTC