Minutes of 2022-06-23 WG meeting is available

Le 28/06/2022 à 15:14, Harald Alvestrand a écrit :
> This recording is now on YouTube:
> https://youtu.be/c3nah_oMZQs

And the associated minutes are available at:

(copied as text below)


                           WebRTC June 2022 VI

23 June 2022

    [2]Agenda. [3]IRC log.

       [2] https://www.w3.org/2011/04/webrtc/wiki/June_23_2022
       [3] https://www.w3.org/2022/06/23-webrtc-irc


           Bernard, Dom, Eero_Hakkinen, Elad, Florent, Guido,
           Harald, Jan-ivar, Patrick_Rockhill, Riju, TimPanton,
           TuukkaT, Youenn


           Bernard, HTA, Jan-Ivar



     1. [4]WebRTC WG Re-Charter
     2. [5]Region Capture Issues
          1. [6]Issue [7]#17: the case for making CropTarget Sync
          2. [8]Issue [9]#17: the case for making CropTarget Async
          3. [10]#17 discussion
          4. [11]Issue [12]#18: Is CropTarget name too generic?
          5. [13]Issue [14]#63: Cropping non-self-capture tracks
          6. [15]Making CropTargets stringifiable
     3. [16]Face Detection
     4. [17]Summary of resolutions

       [7] https://github.com/w3c/mediacapture-extensions/issues/17
       [9] https://github.com/w3c/mediacapture-extensions/issues/17
      [10] https://github.com/w3c/mediacapture-extensions/issues/17
      [12] https://github.com/w3c/mediacapture-extensions/issues/18
      [14] https://github.com/w3c/mediacapture-extensions/issues/63

Meeting minutes

    Recording: [18]https://youtu.be/c3nah_oMZQs

      [18] https://youtu.be/c3nah_oMZQs



    Slideset: [20]https://lists.w3.org/Archives/Public/www-archive/


   [21]WebRTC WG Re-Charter [22]🎞︎

      [21] https://github.com/w3c/webrtc-charter/
      [22] https://www.youtube.com/watch?v=c3nah_oMZQs#t=162

    [23]Draft updated charter for the WebRTC WG

      [23] http://w3c.github.io/webrtc-charter/webrtc-charter.html

    [24][Slide 9]


    dom: we need a new charter, and to discuss [25]issue #70
    (migrating work to another group)

      [25] https://github.com/w3c/webrtc-charter/issues/70

    youenn: would this require rechartering Media WG?

    dom: not sure, possibly

    aboba: we have a joint meeting with Media WG at TPAC

    dom: but we can't wait until then to come to a conclusion

    youenn: migrating items between charters is always challenging
    from our side, in administrative aspecs

    bernard: do we need meetings with the media wg to discuss this?

    dom: if we want to migrate the work, yes - but we first need to
    decide whether we want to do that

    bernard: the fact that we have dependencies on videoframe makes
    it an interesting question to consider

    elad: how busy is the Media WG? would it increase or decrease
    the pace of our work?

    harald: that would have to be something to discuss with the

    dom: bringing it to media wg would bring more of a media
    perspective when this group comes more with a transmission

    harald: so the chairs will discuss this with the Media WG

    dom: no objection from the group in exploring this?


   [26]Region Capture Issues [27]🎞︎

      [26] https://github.com/w3c/mediacapture-region
      [27] https://www.youtube.com/watch?v=c3nah_oMZQs#t=853

     Issue [28]#17: the case for making CropTarget Sync [29]🎞︎

      [28] https://github.com/w3c/mediacapture-region/issues/17
      [29] https://www.youtube.com/watch?v=c3nah_oMZQs#t=853

    [30][Slide 13]


    Jan-Ivar: this is about the cropping API - [31]issue #17 is
    about whether it should be sync vs async
    … long discussion on the issue - I'll be presenting arguments
    why it should be sync
    … The TAG design principles include encouragement to use sync
    APIs when appropriate, with some exceptions (incl cross-process

      [31] https://github.com/w3c/mediacapture-region/issues/17

    [32][Slide 14]


    Jan-Ivar: the API currently in the spec (that doesn't have
    consensus) is async
    … so you have to `await CropTarget.fromElement(element)`
    … I'm proposing it doesn't need to be async,
    … the purpose of the operation is associating an identifier
    with an element
    … as currently specified, it can't fail
    … the goal of the crop target is to share it over postMessage
    across documents

    [33][Slide 15]


    Jan-Ivar: multiple actions needs to happen before we're
    cropping anything
    … cropTarget can be done ahead of time or later
    … if it gets postMessaged to the top-level document, the said
    document can offer to the user to crop to that target
    … it's only at the end of this process that there is a clear
    intent to crop
    … UA can optimize this by running some of the underlying tasks
    early, but that creates risks in case this doesn't go through
    … the complexity of that situation shouldn't be exposed to

    [34][Slide 16]


    Jan-Ivar: [35]#48 is asking to allow failure from the minting
    process due to resource exhaustion
    … which seems linked to optimizations implemented in Chrome
    … The issue is that it allows random document to exhaust
    cropping resources
    … since the API is not gated by permissions - creating DOS
    … and may expose apps that don't deal well with that failure
    … if resource allocation is moved to the cropTo step, this risk
    … similar to mediaSource.getHandle

      [35] https://github.com/w3c/mediacapture-region/issues/48

    [36][Slide 17]


    Jan-Ivar: I believe my proposed API is faster, simpler and
    still optimizable
    … I don't think we need to block on inter-process communication

    [37][Slide 18]


    jan-ivar: failing of optimization shouldn't imply a failure of
    the operation
    … optimizations that influence API design decision tend to
    generate further issues
    … because optimizations create new side effects
    … there is no developer benefits to this API being async
    … and there are general developer costs to async APIs - they
    create pre-emption points which risk data races
    … multiplying failure points for rare error cases is a footgun
    … and async is slower as I've shown

    elad: the TAG offers design principles, but also meta
    principles - not sure it's productive to discuss other browsers
    … the fingerprinting risk is reasonable, but the spec doesn't
    force to surface this
    … a promise doesn't have to fail in your implementation either

    jan-ivar: the API should be designed on principles
    … Mozilla is here to push a better API for the Web

     Issue [38]#17: the case for making CropTarget Async [39]🎞︎

      [38] https://github.com/w3c/mediacapture-region/issues/17
      [39] https://www.youtube.com/watch?v=c3nah_oMZQs#t=2085

    [40][Slide 20]


    Elad: this API is used in production through origin trial
    … we know the API works and makes developers and customers
    … we've learned a lot of lessons by implementing and shipping

    [41][Slide 21]


    Elad: the question is whether minting a token needs to be sync
    or async
    … I'll explain the Chrome decision and that it doesn't impact
    negatively anyone else

    [42][Slide 22]


    Elad: our implementation keeps track of which apps have a crop
    … once it's postMessage'd, this allows Chrome to optimize the
    time-to-cropping when the cropTo method is actually invoked
    … that makes it simpler and more performant, in particular in
    case of CPU congestion in the capturing document

    [43][Slide 23]


    [44][Slide 24]


    Elad: Chrome needs this to be async, whether Mozilla prefers it
    to be sync
    … what's the harm of having an async API?

    [45][Slide 25]


    Elad: Priority of consistuencies goes through users,
    developers, implementers, spec writers

    [46][Slide 26]


    Elad: from a user perspective, seems mostly orthogonal
    … from a developer perspective - what we've heard is that they
    don't care as it doesn't change much in their huge existing
    … implementors - as an implementor, we see this as an
    imperative for us

    [47][Slide 27]


    Elad: what is the negative impact then? Is this theoretical
    … given that IPC is involved, async makes sense

    [48][Slide 28]


    Elad: the TAG actually insists that theoretical purity doesn't
    trump implementers needs

    [49][Slide 29]


    Elad: the TAG discussed this API
    … they were satisfied with the API
    … they haven't seen much ergonomic gain for sync
    … they also highlighted that interop should drive the work of
    the group

    [50][Slide 30]


    Elad: I've shown that consistuencies either don't care about
    sync vs async, and that at least one implementor needs async
    … also, it's easy to go from async to sync, while the other way
    is difficult

     [51]#17 discussion [52]🎞︎

      [51] https://github.com/w3c/mediacapture-region/issues/17
      [52] https://www.youtube.com/watch?v=c3nah_oMZQs#t=2669

    Bernard: by making it async, you're saying that this implies
    that when you have a cropTarget, you know it's ready to use
    … we've had situations in the WebRTC WG where we've found that
    sync APIs needed in fact to be async

    Elad: cf slide 22

    Youenn: I'm surprised you're saying this is MUST - I thought
    this was implementable as a sync API, but that you favored the
    trade-off that async allows
    … as I've pointed out, this trade-off creates a fingerprinting
    and interop issue - so a footgun
    … so I thought both were implementable but sync would be more
    complex in Chrome

    Elad: I don't agree with that characterization

    Youenn: I'm surprised that both approaches are claiming to be
    faster - both can't be true

    Youenn: usually sync APIs are more efficient, except when they
    creating a blocking situation, which I'm not hearing is the
    case here
    … a sync API helps developers, at least a bit

    Elad: resource allocation design decision is orthogonal to sync
    vs async
    … I don't think the resource mitigation limits fingerprinting
    risks can be mitigated through a per-iframe limit
    … in terms of performance, what needs to be fast is cropTo -
    anything before that, the user doesn't notice

    youenn: sync cropTarget minting is faster, but you're saying
    this is not a relevant optimization compared to cropTo

    elad: anything that comes before cropTo is irrelevant to user
    perceived performance (and mostly negligible in any case)

    TimP: as a developer, I have a mild preference to keep it sync
    as it's easier to use
    … I don't think a developer benefit to making it async
    … there may be a user benefit in terms of the crop transition
    … that may convince me of the value if it can be shown
    … from the developer perspective, managing interesting failures
    on obtaining target would also be convincing, but I haven't
    heard that

    Elad: it's a particular choice of trade-offs that require
    async, and async isn't going to harm other consistuencies

    TimP: developers will suffer

    Elad: I claim it's negligible

    TimP: I don't agree - this can generate non-trivial changes,
    although it's certainly doable

    Jan-Ivar: re no downside - nobody claiming that Chrome
    optimizations aren't useful
    … I don't understand why these optimizations can't be done with
    a sync API
    … why can't you implement a fallback?
    … other downsides include fingerprinting, DOS, proliferation of
    async, failure management for developers
    … why does it need to be async?

    Harald: code complexity itself is a risk; this particular
    implementation has been used and tested
    … anyone that depends on cropTo and doesn't notice it failing
    as an issue
    … it's time to stop this discussion - we have seen that Chrome
    claims that a sync implementation would be make it
    significantly more complex
    … the impact on developers is irritating but not fatal
    … we have not seen compelling arguments that we need to change
    what has been proposed
    … I don't see consensus for change, there is an implementation
    of the current spec - I suggest we declare the API to be async
    and move on

    Jan-Ivar: I hear ease of implementation - complexity in the API
    trumps complexity of implementation
    … because of the priority of consistuencies

    Harald: a more complex API that does the right thing is better
    than a simple API that does the wrong thing

    Elad: I'm not seeing consensus, but I don't think there are
    remaining benefits to discuss this

    dom: we could either run a vote, or wait for more
    implementation experience

    TimP: I would be inclined to say that getting other
    implementation is most important
    … sync would be more elegant, but it doesn't look like we're
    going to get that

     Issue [53]#18: Is CropTarget name too generic? [54]🎞︎

      [53] https://github.com/w3c/mediacapture-region/issues/18
      [54] https://www.youtube.com/watch?v=c3nah_oMZQs#t=4111

    [55][Slide 33]


    Youenn: "crop" as at term isn't used too broadly so far, so
    probably OK
    … not sure that "target" helps

    [56][Slide 34]


    [57][Slide 35]


    Youenn: "object whose sole purpose is to be given cropTo" - may
    not be limited to elements
    … the term CropRegion might work better to represent
    … and would align with the spec name ("region capture")
    … thoughts?

    TimP: the name should reflect that it is a token, not a region
    or a target itself
    … if it's opaque, it should say so

    Youenn: it may not remain opaque

    TimP: a region sounds like something you could do math on, e.g.
    calculate its surface area
    … which you can't

    elad: similar reservation - region feels something with
    coordinates; also, a cropTarget isn't static, it can move,
    which cropregion makes more misleading

    harald: this is bikeshedding; I don't see benefit in changing

    Bernard: +1 that CropRegion is confusing; I prefer the current

    Youenn: any interest in clarifying the definition (i.e. whether
    it's a reference to an element or something more generic)
    … I guess that can be done later

    RESOLUTION: close [58]#18 without changing the name of

      [58] https://github.com/w3c/mediacapture-region/issues/18

     Issue [59]#63: Cropping non-self-capture tracks [60]🎞︎

      [59] https://github.com/w3c/mediacapture-region/issues/63
      [60] https://www.youtube.com/watch?v=c3nah_oMZQs#t=4643

    [61][Slide 38]


    Elad: currently you can crop only to current tab
    … I suggest we allow cropping arbitrary tabs

    [62][Slide 39]


    [63][Slide 40]


    Jan-Ivar: a concern is that it might allow sites to censor
    themselves when captured

    Elad: the capturing app can ignore crop targets
    … in fact, cropping automatically would make no sense

    Jan-Ivar: I want the group to be aware of the risk

    Elad: but is it likely?

    Jan-Ivar: I won't predict the future; I don't see other issues
    with this

    harald: please write this up in the github issue; I don't
    understand the risk

    TimP: I support allowing this beyond self-capture

    Elad: can we agree that by next meeting we agree to expand this
    unless a compelling case is made against it?

    Jan-Ivar: imagine a bank wanting to redact what would get
    shared over screen capture
    … can write this up by next meeting

    Harald: I support this too

    dom: me too

     Making CropTargets stringifiable [64]🎞︎

      [64] https://www.youtube.com/watch?v=c3nah_oMZQs#t=5195

    [65][Slide 41]


    Elad: making croptargets stringifiable would help e.g. for
    communication over capture handle
    … not sure I understand the risks

    Youenn: a string makes it much more difficult to
    garbage-collect a croptarget

    Jan-Ivar: +1 to Youenn
    … I haven't heard use cases that justify this
    … having GCable croptarget is good to keep

    youenn: if a croptarget comes with resource allocation, being
    able to end these allocations is a good thing

    elad: you can't just associate the string to the element, and
    when gc'ing the croptarget, remove that association

    TimP: this removes the opacity that I relied in my previous
    … stringifying makes it harder to reason about the safety of

    elad: the only difference between the two is equality

    jan-ivar: there are differences in garbage collection

    TimP: from a developer perspective, there is a difference
    … there are very limited number of paths to get a cropTarget
    … once it's a string, many more paths can be used

    Youenn: can you bring that argument to the github?

    Elad: would like to get resolution to this; we can skip the
    predictable errors

    harald: part of the issue seems to be about reconstructing a
    croptarget from a string (not about stringifying per se)

   [66]Face Detection [67]🎞︎

      [66] https://github.com/w3c/mediacapture-extensions/pull/48
      [67] https://www.youtube.com/watch?v=c3nah_oMZQs#t=5981

    [68]Face Detection explainer

      [68] https://github.com/riju/faceDetection/blob/main/explainer.md

    [69][Slide 45]


    [70][Slide 46]


    riju: this shows our proposal helps reduce power consumption -
    almost 2x at 15fps compared to using TF.js

    youenn: the 1st column has no face detection, and the 2nd is
    doing face detection in the driver?

    tuukka: right

    youenn: in some OSes, the two might be equal if the camera is
    doing it systematically

    riju: indeed, in Android this might be the case

    [71][Slide 47]


    Riju: having a persistant id is very important for face
    … re keeping the probably optional - sure, but all platforms
    provide this
    … a developer may use this to decide to apply further
    processing (e.g. funny hat)
    … but open to making it optional
    … Re VideoFrameMetadata, you suggest coordination on WebCodecs?

    Youenn: yes, we need to engage with them to find the right

    Riju: re API surface, we started with a minimal set, increased
    it based on feedback
    … but we can re-reduce it for the MVP
    … e.g. remove the mesh parts
    … the contour was Harald's request
    … face landmarks are usually important in post-processing,
    think we should keep in MVP

    youenn: my point was in terms of priorities & focus
    … e.g. for the next 6 months

    riju: removing mesh, but keep landmarks

    harald: I still have a problem with the API
    … the power consumption improvement is nice-to-have
    … attachment to the videoframe is nice
    … but still unclear what to use it for
    … the explainer doesn't help much with it
    … what can I do with the output of that API?
    … what the MVP would be viable for?

    riju: e.g. landmarks would be used for post-processing, e.g
    eye-gaze correction
    … the platforms only give bounding boxes at this stage

    Harald: I would like to a more complete use case

    Youenn: one use case is that some encoders optimize based on
    specific bounding boxes

    Bernard: +1 to youenn - segmentation helps with encoding

    Jan-Ivar: the explainer talks about attaching to videoframe,
    but the API is still anchored in mediastreamtrack (e.g. for
    … how would this API be usable on non-camera sources?
    … e.g. on recorded videos

    riju: we couldn't use the same platform APIs to get the power
    consumption benefits

    youenn: I think cameras should be our primary target
    … for recorded videos, you could add this through a media
    capture transform

    jan-ivar: adding these metadata through the transform?

    youenn: yes

    eero: our proposal has support for setting custom metadata in

    harald: the constraints are used to instruct the driver to
    produce the info, which is then attached to the videoframe
    … that makes sense to me
    … but writing up the enhanced encoding use case would help
    making compelling

    riju: any support for prototyping this?

    harald: yes - we need to find compelling applications

    riju: also heard support from youenn

    jan-ivar: I still have some concerns whether this would reveal
    difference across platforms
    … would suggest raising an issue on Mozilla's standard

    bernard: useful to prototype; the metadata discussion should be
    brought to the WebCodecs folks

    riju: will follow up accordingly

Received on Tuesday, 28 June 2022 13:30:50 UTC