Re: Screen sharing and device pixel ratio

Hopefully it's fixed now.  I was subscribed, but had not gone the
permissions-to-archive mail process.

On Thu, Dec 10, 2015 at 1:43 PM, Martin Thomson <martin.thomson@gmail.com>
wrote:

> Mathieu, this seems like a reasonable request.  Can you open an issue
> on the screensharing spec?  Ideally, also a pull request; I'm unlikely
> to get to this soon.
>
> (Yuri, please make sure that you have properly joined the mailing
> list, we're not seeing your mail.)
>
> On 11 December 2015 at 08:28, Mathieu Hofman <Mathieu.Hofman@citrix.com>
> wrote:
> > I’m looking for 2 things right now:
> >
> > -          A getDisplayMedia “constraint” indicating that I would like
> the
> > video stream sized as virtual/logical pixels instead of physical pixels..
> The
> > browser would then internally scale the source content by that ratio.
> >
> > -          Optionally a stream video track attribute, or maybe a stat for
> > the stream, where I could “read” the current devicePixelRatio for the
> stream
> > source, irrespective of whether I have a stream in physical or virtual
> > pixels (the app should know what it asked for)
> >
> >
> >
> > Instead we could maybe extend some of the Web Platform to support video
> > streams with pixelRatios != 1:
> >
> > -          Video MediaStreamTrack would have an internal devicePixelRatio
> >
> > -          HTMLVideoElement would have a videoDevicePixelRatio attribute
> > allowing you to read the video’s device pixel ratio (similar to
> > videoWidth/Height)
> >
> > -          PeerConnection would somehow transmit the devicePixelRatio or
> use
> > that information for its internal down sampling
> >
> >
> >
> > That seems like a much more complicated approach which I’m not sure of
> the
> > implications.
> >
> >
> >
> > Mathieu
> >
> >
> >
> > From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
> > Sent: Thursday, December 10, 2015 13:07
> > To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
> > Cc: public-media-capture@w3.org
> > Subject: Re: Screen sharing and device pixel ratio
> >
> >
> >
> > Okay, so IIUC now, you'd like to know the size, pixel ratio, and perhaps
> the
> > "zoom setting" (affecting stylesheet) of the originally rendered content
> to
> > understand how it was scaled to the video resolution, so that you can
> then
> > post-process for better display downstream.  But, as you mentioned, the
> > source content can change size (e.g., user resizes their window) during a
> > session.  So, this would be per-video frame metadata.
> >
> >
> >
> > I'm not sure what to propose here.  Perhaps others can comment?
> >
> >
> >
> > On Wed, Dec 9, 2015 at 9:40 PM, Mathieu Hofman <
> Mathieu.Hofman@citrix.com>
> > wrote:
> >
> > Resolution constraints don’t help at all. I obviously failed to explain
> this
> > clearly in my original email.
> >
> >
> >
> > There is no way to actually know the size, virtual or physical, of the
> > content to capture:
> >
> > -          It can be an application’s window, or the whole monitor, and
> with
> > proposed selection constraints changes, we wouldn’t even know or control
> > what type of source the user chose to share
> >
> > -          Even if we constrained to screen only somehow, the user can
> chose
> > another monitor than the monitor the browser tab is on, rendering
> > window.screenWidth / screenHeight useless.
> >
> > -          The source can be on a monitor of a different pixel ratio.
> > Windows 10 has independent ratios per monitor.
> >
> >
> >
> > Since we can’t know the size (or it can change dynamically), resolution
> > constraints, fixed or otherwise are useless.
> >
> >
> >
> > Is the problem more clear?
> >
> >
> >
> > From: miu@google.com [mailto:miu@google.com] On Behalf Of Yuri Wiitala
> > Sent: Wednesday, December 9, 2015 21:37
> > To: Mathieu Hofman <Mathieu.Hofman@citrix.com>
> > Cc: public-media-capture@w3.org
> > Subject: Re: Screen sharing and device pixel ratio
> >
> >
> >
> > I just realized I misread the spec.  A ConstrainLongRange extends a
> > LongRange, so you can specify min, max, ideal, and exact for width and
> > height.
> >
> >
> >
> > On Wed, Dec 9, 2015 at 9:27 PM, Yuri Wiitala <miu@chromium.org> wrote:
> >
> > Currently, a fixed resolution can be requested in MediaTrackConstraints..
> > Also, window.screenWidth and window.screenHeight should return the screen
> > size in device-independent coordinates (i.e., always 1440x900 in your Mac
> > Retina example).  Would this accomplish what you need?
> >
> >
> >
> > On Wed, Dec 9, 2015 at 8:40 PM, Mathieu Hofman <
> Mathieu.Hofman@citrix.com>
> > wrote:
> >
> > Hi,
> >
> > The screen sharing spec doesn't mention anything about how to handle
> sources
> > with various device pixel ratios.
> >
> > I believe current implementations simply capture the physical pixels
> without
> > taking into consideration the physical to virtual pixel ratio of the
> source.
> >
> > E.g. if a full screen source of 1440 x 900 has a ratio of 2 dppx (Mac
> retina
> > display), the captured stream will have a size of 2880x1800.
> >
> > As an application I might not want a size that large, especially when
> > transmitting to another display with a different pixel ratio (zoom 200%).
> >
> > Existing max width / height constraints are not adapted to dealing with
> the
> > resolution in this case:
> > - The content is really meant to be scaled by 2, not a floating point
> number
> > that would make it just fit in the constrained size
> > - A small application window might have a size below the max constraints
> >
> > One workaround with the state of the current specs could be to:
> > - gDM
> > - Read width / height of stream
> > - use Window.devicePixelRatio API [1] to guess aspect ratio of content
> > - applyConstraints on stream to scale it by that ratio
> >
> > Unfortunately this might fail on multi-monitor systems that have
> different
> > pixel ratios (Windows 10), when the browser tab and content captured are
> on
> > different monitors. Especially because windows can be dragged across
> > monitors, dynamically changing pixel ratios!
> >
> > What I would like is a sort of "constraint" on the gDM call to indicate
> to
> > the browser I need a stream "scaled" to be in virtual pixels instead of
> > physical pixels.
> >
> > Additionally, I would also consider adding a way to "read" the current
> pixel
> > ratio of the source. This is to know if the captured stream is currently
> > scaled when I ask for a "virtual pixels" stream, or the factor by which
> to
> > scale it when asking the "physical pixels" and I want to do that
> processing
> > on my own (e.g. I want to compute a scaling factor adapted to the pixel
> > ratio of the receiving display)
> >
> > Would there be any issues adding these to the spec?
> >
> > Mathieu
> >
> > [1] https://drafts.csswg.org/cssom-view/#dom-window-devicepixelratio
> >
> >
> >
> >
> >
> >
>