Re: [mediaqueries] Media Source Extensions and device capabilities

Hi Tab,

I am not really a CSS expert, so that is why I posted the underlying
requirement without presuming a CSS solution.

But, naively, I would think that if we had a "device pixels" unit (dpx)
corresponding to physical device pixels, we could construct a media query
along the lines of "min-height: 1080dpx". This could be used to annotate a
<source> element on an HTMLMediaElement or with window.mediaMatch() to
obtain events as the window is moved to / from displays with HD resolution.

Likewise, one could imagine, "min-audio-channels: 6", although as per the
previous discussion of this there are some details to work out as to
exactly what is meant by "audio channel" here. As with video, the
requirement is to avoid *where possible* delivering media which is just
going to be down-mixed or down-scaled. So, for audio, the
"min-audio-channels: 6" might fail to match only if the UA knows for sure
that 6-channel audio will be down-mixed.

...Mark


On Tue, Sep 16, 2014 at 10:31 AM, Tab Atkins Jr. <jackalmage@gmail.com>
wrote:

> On Tue, Sep 16, 2014 at 8:41 AM, Mark Watson <watsonm@netflix.com> wrote:
> > All,
> >
> > Some time ago I raised on this list the idea that Media Queries should
> > include some capabilities (such as native screen resolution, audio output
> > channels) that might assist Javascript in selection of appropriate media
> to
> > feed into the Media Source Extensions to the HTML Media Element [1]. For
> > example the mediaMatch() function could be used to query device
> capabilities
> > and to receive events in response to changes in device capabilities.
> >
> > Since media queries are what are used declaratively for resource <->
> > capability matching on the HTML Media Element when using static
> resources,
> > it seems natural to also use them for the Media Source Extension case.
> >
> > At the time, the response was that resource selection should be
> performed by
> > the UA, not the author. For adaptive streaming, this option has been on
> the
> > table for a long time: various manifest formats exist which can describe
> > available equivalent media streams (DASH, HLS, Smooth Streaming) and UA's
> > *could* support these, giving complete control of the stream selection to
> > the UA.
> >
> > However, in practice, only one desktop UA has supported any such format,
> and
> > it is a proprietary one (Safari and HLS). All desktop UAs are working on
> -
> > and in some cases have deployed - the alternative Media Source Extensions
> > model. That specification is close to CR. At least the two largest video
> > sites in the US have adopted this approach and have large-scale deployed
> > services.
> >
> > Whilst this approach allows the site to adapt the streamed media to
> > available bandwidth and certain device capabilities exposed by
> canPlayType
> > and media queries, we are still missing the capability to adapt to
> display
> > resolution and audio output capability.
> >
> > These are optimizations and cannot be done perfectly. For example, a
> display
> > may be HD but the video may be in a small window with less than SD
> > resolution. Or a device may have a digital audio output connected, but
> it's
> > connected to an amplifier that performs down-mixing to stereo.
> >
> > Nevertheless, the optimizations which can be performed become quite
> valuable
> > when considering UHD and high quality multi-channel audio. Optimizing for
> > display capability not only saves bandwidth, but may avoid the need for
> > downscaling, saving battery life.
> >
> > In discussions in the html-media group, it's been suggested that this
> topic
> > should be the subject of a general solution, not something specific to
> MSE.
> >
> > Is there any interest in re-visiting this topic here ?
> >
> > Regards,
> >
> > Mark Watson
> >
> > [1]
> >
> https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html
>
> I'm interested in exploring this topic more, certainly.  Can you give
> some markup example of how a new MQ would be used in this context?
>
> ~TJ
>