Re: [mediaqueries] Media Source Extensions and device capabilities

On Tue, Nov 18, 2014 at 9:45 AM, Tab Atkins Jr. <jackalmage@gmail.com>
wrote:

> On Tue, Nov 18, 2014 at 9:06 AM, L. David Baron <dbaron@dbaron.org> wrote:
> > On Tuesday 2014-11-18 14:59 +0100, Florian Rivoal wrote:
> >> > On 14 Nov 2014, at 18:13, L. David Baron <dbaron@dbaron.org> wrote:
> >> > Then again, I probably wouldn't be against just having a separate
> >> > unit for device pixels (or, for that matter, for true lengths),
> >> > although I'm not sure I like 'dpx' as the name (looks too much like
> >> > 'dpi’).
> >>
> >> We already have dppx (Dots Per PiXels) for the resolution MQ, to query
> the number of physical pixels in a css pixels, so I don’t think dpx is a
> name I’d recommend.
> >>
> >> As for physical units for lengths, coming from mostly anyone else, I’d
> probably reply with the usual story about why this is misguided, but as you
> know all these already, I am curious as to why you think this is a
> potentially good idea.
> >
> > I think it's a potentially good idea here because what we're talking
> > about is media resource selection.
> >
> > Consider the problem as follows:  the server can offer a video at
> > four different resolutions, ranging from, say, 320x180 up to
> > 1280x720.  (I'm making up these details, but this is the general
> > idea.)  They want to offer the highest quality one that's going to
> > be useful, but don't want to waste bandwidth sending the high
> > quality one to a small display.
> >
> > So the right choice of video is a function of how many *device
> > pixels* wide and tall the space for the video is.  It's not a
> > function of CSS pixels; treating it as such would send low quality
> > video to displays with more than one device pixel per CSS pixel.
>
> This problem seems identical to that of <picture>, and particularly
> the sizes/w syntax, where you don't know precisely how wide, in CSS
> px, the image will end up being displayed, so you can't tell what the
> density of any given source file will be (since the density is image
> pixels / CSS px).
>
> The solution is not to use MQs, because that requires a bunch of cases
> (which multiply as you have different sizes), and requires guessing at
> what point switching over to a different video source might be better.
> (We have some examples of the kind of markup required to serve a
> couple of different image sources to devices adequately.  It's... not
> pretty.)  The right solution is to tell the browser how many pixels
> wide the source is, and how wide it'll end up, so the browser can
> figure out the effective density for you, and choose the correct
> source.
>
> This might be something we need to solve in HTML, by adopting
> something analogous to srcset.  Or, if we're okay with letting CSS
> override <video> source selection, we can solve it with something
> parallel to image-set(), specialized for videos.
>

​I think it's clear that for the <source> / @srcset case, then giving the
UA the information is needs to make the choice is the right solution.

For our use-case, though, we are using the Media Source Extension so it is
fundamentally the script that decides which stream is downloaded. So the
script needs the same information the UA would use in a declarative setting.

Media Queries might not be the right solution, it was just an idea.

A couple of things to bear in mind:
- at least for our use-case we are expecting the video to mostly be in
fullscreen mode. So that is the case we care about most. How it should work
when the video is not in full-screen is not something I have strong
opinions on.
- for streaming services with MSE, we are typically buffering several
seconds or minutes ahead of playback. As a result, the kind of device
restrictions we're interested in adapting to are the long-lived ones, such
as physical display resolution. If the video is displayed in a small window
but the user might go full-screen at any time, we might prefer to deliver
the high resolution version all the time "just in case"

​...Mark



>
>
> Audio output capability is still something that can be addressed via
> MQs.  (Though it sounds like offering the number of audio channels
> directly via some JS API would also be useful to Mark Watson's
> use-case.)
>
> ~TJ
>