FW: [Minutes] Media Working Group Teleconference - 2020-07-14

Cross-posting for those who access CSS minutes from this list.

On 7/20/20, 5:25 AM, "Francois Daoust" <fd@w3.org> wrote:

    Hi all,

    The minutes of last week's joint meeting between the Media WG and the 
    CSS Media WG are available at:

    The groups discussed solutions and problems to expose bi-plane 
    (video/graphics) capabilities to applications. Minutes are copied as raw 
    text below.


    Joint meeting between CSS WG and Media WG
    14 July 2020

        [2]Agenda. [3]IRC log.

           [2] https://github.com/w3c/media-wg/issues/18

           [3] https://www.w3.org/2020/07/14-mediawg-irc


               Alan Stearns, Barbara Hochgesang, Becca Hughes, Chris
               Needham, Daniel Holbert, Florian Rivoal, Gary Katsevman,
               Greg Freedman, Jer Noble, Jimmy Huang, Kevin Babbitt,
               Matt Wolenetz, Mounir Lamouri, Nigel Megitt, Peng Liu,
               Rossen Atanassov, Simon Fraser, Simon Thompson, Theresa
               O'Connor, Vi Nguyen


               Jer, Mounir

               Jer, Mounir, Vi


         1. [4]Introductions
         2. [5]Video Media Queries

    Meeting minutes


          * Florian Rivoal, Invited Expert
          * Daniel Holbert, Mozilla
          * Alan Stearns, Adobe
          * Matt Wolenetz, MSE spec co-editor, Google
          * Jer Noble, Apple
          * Greg Freedman, Netflix
          * Simon Fraser, Apple
          * Rossen Atanassov, Microsoft
          * Jimmy Huang, Intel
          * Mounir Lamouri, Google
          * Kevin Babbitt, Microsoft
          * Nigel Megitt, BBC, Chair TTWG
          * Simon Thompson, BBC
          * Theresa O'Connor, Apple
          * Vi Nguyen, Microsoft, Media Capabilities
          * Peng Liu, Apple
          * Chris Needham, BBC, Media & Entertainment IG co-chair
          * Gary Katsevman, Brightcove, Chair TTWG
          * Chris Cunningham, Google

       Video Media Queries

        Related issues:
          * [6]Media WG issue #135 — hdrSupported - Screen.video or
            simply Screen?

          * [7]CSS WG issue #4471 —
            [cssom-1][mediaqueries-5][css-color] Dealing with bi-plane
            (video/graphics) when reporting values

           [7] https://github.com/w3c/csswg-drafts/issues/4471

        florian: introduction of topic: some devices have separate
        planes for video and graphics contents

        florian: these planes have separate properties including width,
        height, and resolution. The existing queries target only the
        graphics plane

        florian: video sites care mostly about the graphics plane, and
        would like api to choose an appropriate variant

        florian: originally the CSS WG accepted this proposal, but
        found issues and wondered if these were the correct way of
        solving this problem

        florian: Media Queries need to come with a unit; CSS pixels?
        CSS Pixels can mean an angular measurement that changes size
        based on the distance from the screen. Physical pixels? anchor
        a CSS measurement to a physical measurement.

        florian: for the video plane, you may want actual literal
        pixels, which would include a third definition.

        florian: One possibility would be to make these properties

        florian: this issue has an analogue for images, and the general
        path has been not to do this through Media Queries.

        florian: images must also be sensitive to changes in resolution
        as windows move between windows, but also to available

        florian: for images, the UA will choose the appropriate image
        variant based on not only resolution, but other factors.

        florian: this may not be the same solution available for video,
        but there could be a .js API in which the page provides the
        available resolutions and properties, then the UA chooses the
        correct variant

        florian: were there other arguments pro or con for Media

        chcunningham: question: does it not solve the problem by
        mapping the pixel ratio to a fixed value?

        florian: we could make an allowance, but it's not currently

        florian: what does that then mean when people query for the
        video plane in "em" units?

        florian: There's not a number that is based on font sizes that
        is correct.

        chcunningham: if you use the existing mechanism that ratio, you
        could discover the ratio.

        florian: Media Queries don't allow you to query for the size of
        the screen, it only allows you to query for whether the screen
        is within a provided value

        florian: maybe this is the wrong tool, since you can't query
        the value directly.

        chcunningham: I don't think that's a dealbreaker, since sites
        will have a list of formats or variants with specific
        resolutions, so that works with the way Media Queries work

        chcunningham: w.r.t. the Picture API; this would be a profound
        change from how media APIs are used on the web.

        chcunningham: in MSE and WebCodecs, the page has low level
        access to variant selection

        chcunningham: clients prefer to make these decisions themselves

        dholbert: there are also "inch' and "centimeter" units

        dholbert: that would be something difficult to define; argument
        for unitless value

        florian: if we wanted to do this, we don't want to break all of
        CSS units, so it would still be a fixed ratio between this
        value and a CSS inch

        cpn: There was a proposal to expose these values on the screen
        object; is there a value to re-visiting that decision?

        florian: the query example is only one possibility; we could
        also just expose those numbers explicitly.

        cpn: a proposal which inverts the current model that gives all
        the variant selection to the client would be a big change

        wolenetz: is there already a way for JS apps to determine the
        resolution where the video will be composited?

        florian: I think the Media Queries was proposed as a way to
        expose that value.

        wolenetz: the current way media works on the web is to expose
        to clients underlying capabilities and allow the site to choose

        Simon_T_BBC: set top boxes should be thought about.

        Simon_T_BBC: HDMI capabilities must also be taken into account;
        some may only be capable of a certain frame rate

        Simon_T_BBC: There are some non-screen parameters that are
        important for deciding which variant to choose.

        florian: you may want to make a different tradeoff than just
        available resolution

        jernoble: as the person who proposed MediaQueries as a possible
        solution. Media Queries already provides event mechanism for
        when that ratio changes when moving windows to different
        screens. If we did expose something to Screens API, we have to
        figure out events

        chcunningham: I understand CSS WG is concerned that this is not
        the correct solution. But we ran this by the WG before it was
        proposed. Are we back to the drawing board, or is this a minor

        florian: I don't think the CSS WG has concluded that this was
        impossible; merely uneasy. On further reflection, the way it is
        spec'd cannot work. It doesn't work for print media. We would
        need to fix the unit issue to make this possible.

        florian: we could fix this, but the questions this raises makes
        this solution feel "odd".

        florian: there's also a desire in the CSS WG that for devices
        that do not have a separate video plane, these properties
        should be the same for the graphics plane.

        florian: screen properties were deprecated, because leaking
        non-important information, and those properties have privacy

        florian: CSS pixels can and will be used incorrectly

        florian: we think this should return to the drawing board.

        florian: the current approach is possible, but not ideal.

        ving: the Media WG did discuss exposing this to the Screen
        object; could we get a summary of why Screen is not the ideal

        ving: also, there was a concern about interfaces and cables,
        but that should be handled through MediaCapabilities API. We're
        trying to solve the problem of exposing the exact resolution of
        the video plane on the actual display.

        chcunningham: where are you seeing these concerns jernoble

        ving: in the notes.

        chcunningham: I don't oppose putting this on screen; florian
        what do you think of having device pixels on the Screen?

        florian: my personal feeling is that's ok; we'd need events if
        that changes

        florian: is this a concern for those devices that have a
        separate video plane?

        mounir: I don't think an event is really an issue. The Screen
        object doesn't have an event. This is an existing problem in
        that the Screen's properties can change without an event.

        chcunningham: if we expose device pixels on the screen object,
        we're hitting the same reasons why the existing properties are

        florian: if we expose the video device pixel, you'd just get a
        number that is in device pixels. You wouldn't have a way to ask
        for the video plane in "em" unit.s

        florian: yes, we'd still have the privacy concern.

        GregFreedman: if you put device pixels on the Screen object,
        you'd still have the problem of the difference between the
        video and graphics plane. Would you have device pixels for both

        GregFreedman: I kind of liked where we landed with the Media
        Queries. what we want to do is say: can you support this
        resolution on the video plane; and that seems like this works
        with Media Queries well.

        GregFreedman: is the problem that CSS doesn't like the unitless

        florian: yes, CSS as a language could support it, but if we're
        on a device that does not have a separate plane, we probably
        want width: to do the same thing as video-width:

        florian: additionally, that definition also affects print media

        florian: so what should you do on print media?

        <Zakim> nigel, you wanted to ask for a reminder of the use
        cases that remain if we use MediaCapabilities to choose which
        video to get

        nigel: the use case is originally to detect what variant to
        display. If MediaCapabilities is the best way to query for
        that, what's left in CSS ?

        GregFreedman: the device may be capable of doing 4k UHD, but if
        the display is only 1080p SD, there's no reason to pick that
        variant. so we split it into two APIs: one to query decode, one
        to query display.

        ving: MediaCapabilies also is not screen aware.

        nigel: could Media Queries handle that scenario, where there's
        multiple displays?

        hober: we already handle that with the 50%+1 to determine which
        monitor's properties to return.

        chcunningham: that's the implementation in chrome.

        cpn: can the UA play this particular content is answered by the
        combination of screen and decode capabilities.

        cpn: asking these two questions separately doesn't give a
        complete picture of whether a variant can be successfully

        GregFreedman: Media Capabilities (MC) asks "can we decode this"
        and Media Queries (MQ) asks "should we decode this".

        nigel: is the set of things we need to know just the video
        plane resolution, and the frame rate?

        nigel: are there other properties that we need to include?

        florian: those do not yet exist?

        <Zakim> wolenetz, you wanted to say that decoder capabilities
        exceeding display capabilities shouldn't limit what is

        nigel: just wondering about design and what info needs to be

        wolenetz: Media Capabilities already gives you the answer of
        whether you can decode a given piece of media. Do we try to
        move some of the properties in Media Capabilities into Media

        mounir: oginially there was information in the Screen object.

        wolenetz: since Media Capabilities is already being asked about
        candidates, maybe Media Capabilities is the best place to ask
        more questions

        wolenetz: since I'm already using Media Capabilities, doing
        that again against a different API is difficult.

        chcunningham: I have the exact opposite opinion, respectfully.
        the Screen API and the Media Queries API already exists.

        chcunningham: we never considered putting Media Capabilities
        queries into the Screen API

        mounir: websites already will react to window size changes,
        e.g., when going into fullscreen.

        jernoble: we want to answer the question whether the video
        should be upscaled or downscaled

        jernoble: so maybe we should design an API for that?

        smfr: I would be reluctant to add new APIs to the Screen api
        due to fingerprinting concerns.

        <Zakim> florian, you wanted to suggest that the JS API could
        take the frame rate as input more easily than Media Queries

        florian: if the frame rate matters, it would be more amenable
        to a Screen API, since you can pass in parameters to that API,
        but that style of query isn't possible with Media Queries.

        chcunningham: no option seems to be the clear winner; IMO the
        least of several evils was the CSS Media Queries options.

        <nigel> CSS Media Queries don't remove the fingerprinting

        chcunningham: I wonder if there is a way to fix this problem

        florian: I'd like to ask Simon to clarify this answer about

        smfr: it depends on whether it allows you to enumerate all
        screens. and depends on whether the API allows you to detect
        whether users have changed their screen resolution or are using
        non-standard third party screens

        smfr: it's still doable with Media Queries, but you have to
        binary search, so it's not as easy.

        mounir: what's the next step here? shall we move this to GitHub

        chcunningham: we haven't discussed any options beyond "fix
        Media Queries" or "move to Screen".

        <astearns> lost my webex connection - would there be a way of
        making a video-playback-specific API that would expose less
        fingerprinting surface but still be able to answer coarse
        questions on what can be displayed? How fine-grained do we
        really need to be for playback?

        chcunningham: maybe now that we've had this meeting the CSS WG
        will come up with an alternate solution that better solves all
        the use cases

        mounir: lets follow up on GitHub; lets try to discuss between
        MediaWG and CSS WG people there

        florian: it's unfortunate that there were fewer voices in the
        CSS WG than I anticipated; there are conflicting desires in the
        CSS WG.

        florian: I'm unable to represent all those various POV.

        florian: but yes, we can explore; either alternative ways of
        speccing the Media Queries or alternative API entirely.

        chcunningham: thank you florian! and thank you for representing
        the CSS WG.

Received on Monday, 20 July 2020 16:14:00 UTC