W3C home > Mailing lists > Public > public-webrtc@w3.org > October 2022

[minutes] Recording of October 18 meeting

From: Dominique Hazael-Massieux <dom@w3.org>
Date: Tue, 25 Oct 2022 19:17:31 +0200
Message-ID: <38580589-935b-e1ab-147a-44c4e4fd9e33@w3.org>
To: "public-webrtc@w3.org" <public-webrtc@w3.org>
Le 25/10/2022 Γ  18:39, Harald Alvestrand a Γ©critΒ :
> The recording of the October 18 WG meeting is here:
> https://youtu.be/cbLpKnU6RoI

And the minutes (which incorporate the video) at 
https://www.w3.org/2022/10/18-webrtc-minutes.html (copied as text below).


                       WebRTC October 2022 meeting

18 October 2022

    [2]Agenda. [3]IRC log.

       [2] https://www.w3.org/2011/04/webrtc/wiki/October_18_2022
       [3] https://www.w3.org/2022/10/18-webrtc-irc


           Ben, Bernard, Dom, Elad, fippo, Florent, Harald, Henrik,
           Jan-Ivar, MikeEnglish, PatrickRockhill, Tony, Tove,
           Varun, Youenn


           Bernard, HTA, Jan-Ivar



     1. [4]Encoded Transform - Overflow from TPAC
          1. [5]Issue [6]#109 & [7]#119 Depacketization order
          2. [8]Issue [9]#143 generateKeyFrame
          3. [10]Issue [11]#158 / [12]PR #140: add mimeType to
          4. [13]Issue [14]#154: add rtp seqNum to inbound audio
          5. [15]Issue [16]#131: Packetization API
     2. [17]Media Capture Extensions
          1. [18]PR #77: Add MediaStreamTrack framesCaptured and
     3. [19]WebRTC & Simulcast
          1. [20]Issue [21]#2732: Inconsistent rules for rid in
          2. [22]Issue [23]#2764: What is the intended behavior of
             rollback of remote simulcast offer?
          3. [24]Issue [25]#2737 / [26]PR #2788: Modifications to
             [[SendEncodings]] from setParameters and sLD/sRD can
             be racy
          4. [27]Issue [28]#2762: Simulcast: Implementations do not
             fail (and that seems good)
     4. [29]WebRTC Extensions: Data Channels
          1. [30]Issue [31]#114: RTCDataChannel transfer and
          2. [32]Issue [33]#115: Need to specify behavior of
             detached RTCDataChannel objects
     5. [34]Capture Handle
     6. [35]Summary of resolutions

       [6] https://github.com/w3c/mediacapture-handle/issues/109
       [7] https://github.com/w3c/mediacapture-handle/issues/119
       [9] https://github.com/w3c/mediacapture-handle/issues/143
      [11] https://github.com/w3c/mediacapture-handle/issues/158
      [12] https://github.com/w3c/mediacapture-handle/pull/140
      [14] https://github.com/w3c/mediacapture-handle/issues/154
      [16] https://github.com/w3c/mediacapture-handle/issues/131
      [18] https://github.com/w3c/mediacapture-handle/pull/77
      [21] https://github.com/w3c/mediacapture-handle/issues/2732
      [23] https://github.com/w3c/mediacapture-handle/issues/2764
      [25] https://github.com/w3c/mediacapture-handle/issues/2737
      [26] https://github.com/w3c/mediacapture-handle/pull/2788
      [28] https://github.com/w3c/mediacapture-handle/issues/2762
      [31] https://github.com/w3c/mediacapture-handle/issues/114
      [33] https://github.com/w3c/mediacapture-handle/issues/115

Meeting minutes

    Recording: [36]https://youtu.be/cbLpKnU6RoI

      [36] https://youtu.be/cbLpKnU6RoI



    Slideset: [38]https://lists.w3.org/Archives/Public/www-archive/


   [39]Encoded Transform - Overflow from TPAC [40]🎞︎

      [39] https://github.com/w3c/webrtc-encoded-transform
      [40] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=152

    Harald: during TPAc, we discussed the concept of a packet API,
    with an explainer, use cases and architecture - not yet done
    … other issues didn't get covered

     Issue [41]#109 & [42]#119 Depacketization order [43]🎞︎

      [41] https://github.com/w3c/webrtc-encoded-transform/issues/109
      [42] https://github.com/w3c/webrtc-encoded-transform/issues/119
      [43] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=202

    [44][Slide 12]


    Harald: packets don't arrive order on the network (they get
    lost or retransmitted)
    … frames need to be in order for the decoder
    … in general, a transformation is simpler when happening in
    decoding order
    … this requires a jitter buffer in front of the decoder
    … if the transformer itself introduces jitter, it doesn't get
    … currently Chromium has the jitter after the transformer

    Bernard: this isn't the only place where we're encoutnering
    this problem
    … are you imagining an explicit API for jitter buffer - e.g. a
    jitter buffer provided as a transform stream?

    Harald: we could say "frames arrive in the order they arrive
    in", vs "the UA reorder them, incl waiting for frames"
    (probably not good), with a flag allowing one or the other

    Youenn: I recall we discussed this previously
    … iirc, we thought that in-order matched the Web developers
    … it may make it harder to implement for UA
    … we should look at use cases where having out-of-order would
    be a benefit
    … it's a possible footgun; if there are good use cases for it,
    then we should look for a solution, but otherwise, we should
    stick with in-order as in the spec

    Bernard: for the crypto use case, is out-of-order even doable?

    Youenn: for SFrame yes
    … the counter may not be monotonic in that situation
    … which would lead to dropped frames
    … but it shouldn't be an issue from a decryption perspective

    Bernard: so is out-of-order a speed concern?

    hta: my worry about is in-order is in the case of lost frames
    … without nack, rtx - you have to give up at some point
    … if we accept in-order frames, we accept that lost frames will
    cause delays of some magnitude

    dom: the wait-for-loss delay could be provided by the

    youenn: having both options would create complexity for
    … if the transform is taking sometimes 2ms and sometimes much
    longer, a jitter buffer would then be beneficial
    … for decryption or metadata passing, it should be fairly
    … not sure of the value of a jitter buffer positioned after

    hta: sounds like we need more time on use cases

    Tony: moving the jitter buffer earlier means increased packet
    loss (given that it removes the processing time from the jitter
    … there will be delays introduced from operating in a worker
    (rather than say a real time worklet)

    youenn: currently chrome & safari implementations do
    out-of-order, which don't match the spec
    … is Chrome planning to move to in-order? if implementations
    don't intent to align with the spec, that's also a

    hta: switching to in-order would require a compelling argument

    jib: unless the transform has side effects (time-dependent), it
    shouldn't matter too much
    … use cases would be helpful
    … out-of-order seems a footgun - why should developers worry
    about that?

    hta: if delay matters, in-order is a footgun

    youenn: so we should use cases for both in-order and

     Issue [45]#143 generateKeyFrame [46]🎞︎

      [45] https://github.com/w3c/webrtc-encoded-transform/issues/143
      [46] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=984

    [47][Slide 13]


    [48]TPAC discussion

      [48] https://www.w3.org/2022/09/13-webrtc-minutes.html#t07

    Fippo: I wanted to suggest a 4th proposal - an empty return
    value, but allow the app to pass any subset of the rids to
    generate keyframes
    … some encoders can generate keyframes from individual rids,
    others can't - it depends on the codecs

    hta: the argument list API would thus be strictly more powerful
    without additional implementor burden

    youenn: at TPAC our conclusion was one rid was good & simple
    enough; we didn't have use cases for 2 layers hitting the same
    … an encoder-behavior dependent API isn't so helpful, but I
    agree it isn't a big burden to add either

    hta: medium objection to single value, no strong objection to
    array - should we go with the array args?

    RESOLUTION: pass an array arguments to generateKeyframes

    fippo: I'll do the PR

     Issue [49]#158 / [50]PR #140: add mimeType to metadata [51]🎞︎

      [49] https://github.com/w3c/webrtc-encoded-transform/issues/158
      [50] https://github.com/w3c/webrtc-encoded-transform/pull/140
      [51] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=1282

    [52][Slide 14]


    HTA: figuring the meaning of a payload requires parsing the SDP
    to figure out what was negotiated
    … the UA already knows which mime type is associated with which
    payload type

    Fippo: another argument for it is that we don't specific how
    the data is structured
    … being able to specify it as depending on the mime type would
    be good

    youenn: thanks, this provides a good use case
    … I think that's a pattern we already apply elsewhere

    Fippo: in stats, indeed

    Florent: isn't that available via getParameters? that exposes
    the list of payload types

    HTA: but only if you have the PC

    Fippo: that's harder in workers

    RESOLUTION: Add mimeType to metadata

     Issue [53]#154: add rtp seqNum to inbound audio [54]🎞︎

      [53] https://github.com/w3c/webrtc-encoded-transform/issues/154
      [54] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=1451

    [55][Slide 15]


    Fippo: we have a custom decoder that relies on the rtp sequence
    number to detect loss in the audio
    … relatively easy to add to incoming frames for audio
    … more complicated for video, or for outgoing frames

    HTA: for incoming audio, you have one packet resulting in one
    set of samples

    youenn: coming back to in/out-of order, this would expose that
    … if we're not doing in-order, this may create confusion

    Fippo: in our use case, we have our custom JS jitter buffer; we
    don't reenqueue the frame into the pipeline

    HTA: so that's also a use case for out-of-order: bring your own
    jitter buffer

    Fippo: I can that written up as input to the other discussion

    HTA: are we happy to expose this only for audio incoming
    frames, as a non required dictionary?

    jib: I think it would still be interesting to understand better
    this one-ended use cases

    HTA: ok, so let's wait for the use cases before proceeding then

     Issue [56]#131: Packetization API [57]🎞︎

      [56] https://github.com/w3c/webrtc-encoded-transform/issues/131
      [57] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=1717

    [58][Slide 16]


    HTA: any more comment on the packetization API beyond what was
    discussed at TPAC?

    Youenn: we could start with things like MTU

    HTA: in the frame API?

    Fippo: MTU is mostly an issue for audio; I don't think we hit
    that threshold even with redundancy
    … it becomes an issue with transform that changes size largely

    Youenn: I don't think adding the MTU to the frame API would
    make sense - more at the context level, with changes signaled
    via events
    … the frame is coming from the encoder, that's not where the
    MTU info lives

   [59]Media Capture Extensions [60]🎞︎

      [59] https://github.com/w3c/mediacapture-extensions/
      [60] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=1878

     [61]PR #77: Add MediaStreamTrack framesCaptured and framesEmitted

      [61] https://github.com/w3c/mediacapture-extensions/pull/77
      [62] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=1884

    [63][Slide 18]


    Henrik: `track.getSettings().frameRate` tells the configured,
    but not actual frame rate
    … knowing the actual frame rate and the dropped frames would be
    … some of that are exposed in stats, or in media playback
    … but the measurements are happening later in the pipeline -
    e.g. if the frame is dropped as soon as it is produced, it
    won't show up
    … and we shouldn't force a webrtc PC to get track specific info

    [64][Slide 19]


    henrik: my proposal is to add a frame counter to track API,
    with a `getStats()` method

    youenn: all APIs that are using an MST will allow you to get
    the number of frames that you're actually receiving
    … Media capture transform gives you the count of frames,
    likewise for WebRTC & HTMLMediaElement
    … what you want is focused between the sink & the source
    … not sure I understand the diff between emitted and captured -
    that feels a bit specific to a specific pipeline
    … in our model, it's not clear it would be easy to specific an
    interoperable way to distinguish captured from emitted
    … so maybe focusing first on captured?

    henrik: that makes sense; captured is the main gap in any case

    jan-ivar: framesCaptured makes sense with a low-lighting camera
    use case (although we could revisit the constraint model for
    … share Youenn's concerns for emitted, which feels
    implementation dependent
    … I'm not sure about `getStats()` vs a constraint

    Bernard: next step?

    Henrik: I'm hearing support for framesCaptured in some form,
    and leave emitted for later

    HTA: framesEmitted makes sense for consistency, but I see the
    argument that it may be redundant
    … so let's start with framesCaptured as accepted

    RESOLUTION: move forward with framesCaptured only for now

   [65]WebRTC & Simulcast [66]🎞︎

      [65] https://github.com/w3c/webrtc-pc/
      [66] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=2373

     Issue [67]#2732: Inconsistent rules for rid in
     RTCRtpEncodingParameters [68]🎞︎

      [67] https://github.com/w3c/webrtc-pc/issues/2732
      [68] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=2377

    [69][Slide 23]


    jib: following up to our discussions started in TPAC about rid
    … limiting RID length to 16 characters would help with web
    … an errata has been published on RFC8851 removing - and _
    … feedback on restricting the length would be hard as an
    erratum, but could be done in a -bis

    hta: note that the empty string is outlawed by the BNF

    dom: if we wait for -bis, are implementations going to be
    updated to match the allowed lengths?

    florent: it should be possible to update chrome in that
    direction if we think if it's a good idea

    hta: we don't know of any use case where 17 characters are

    youenn: we could limit to 16 characters with a note mentioning
    ongoing IETF discussion

    jib: we could also have a separate decision on addTransceiver
    vs accepting incoming offers and answers

    dom: I don't think it goes against the protocol to limit what
    the API accepts to generate rids (we should definitely accept
    any valid rid in O/A)

    jib: but then you have an API that doesn't let you set values
    that you accept from a remote description

     Issue [70]#2764: What is the intended behavior of rollback of remote
     simulcast offer? [71]🎞︎

      [70] https://github.com/w3c/webrtc-pc/issues/2764
      [71] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=2850

    [72][Slide 25]


    RESOLUTION: proceed with the proposed clarification

     Issue [73]#2737 / [74]PR #2788: Modifications to [[SendEncodings]]
     from setParameters and sLD/sRD can be racy [75]🎞︎

      [73] https://github.com/w3c/webrtc-pc/issues/2737
      [74] https://github.com/w3c/webrtc-pc/pull/2788
      [75] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=2991

    [76][Slide 26]


    hta: should that addition also be guarded by "if remote is

    jib: it would have to also have a "is an answer" gate - I can
    update the PR

    henrik: if you restart and apply the steps again, wouldn't you
    implicitly rollback anything changed by the in-parallel
    … to do that correctly, you would have to wait until the SDP is

    jib: this is run before we call the success callback
    … we would wait until all setParameters are settled
    … similar to if a remote description came right after

    henrik: so this is done before the SDP process?

    jib: right

     Issue [77]#2762: Simulcast: Implementations do not fail (and that
     seems good) [78]🎞︎

      [77] https://github.com/w3c/webrtc-pc/issues/2762
      [78] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=3270

    [79][Slide 27]


    [Varun, Youenn depart]

    [80][Slide 28]


    RESOLUTION: close [81]#2762 as is

      [81] https://github.com/w3c/webrtc-pc/issues/2762

   [82]WebRTC Extensions: Data Channels [83]🎞︎

      [82] https://github.com/w3c/webrtc-extensions
      [83] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=3577

     Issue [84]#114: RTCDataChannel transfer and maxMessageSize [85]🎞︎

      [84] https://github.com/w3c/webrtc-extensions/issues/114
      [85] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=3586

    [86][Slide 32]


    florent: RTCDataChannels are transferable; maxMessageSize in
    RTCSctpTransport needs to be checked before sending data over a
    … with a channel transferred to a worker, the maxMessageSize
    may be renegotiated on the main thread, which wouldn't be
    visible to the worker trying to send data

    [87][Slide 33]


    Florent: we could prevent changing the maxMessageSize during
    renegotiation - doesn't really happen in practice
    … then that value could be kept in the transferred
    rtcdatachannel and keep the send algorithm as is
    … the other aspect to consider is that the datachannel might
    have been transferred before the initial negotiation
    … updating that value of maxMessageSize could be done as part
    of the "announcing a data channel as open" algorithm

    dom: how confident are we that maxMessageSize can be frozen in
    renegotiation is web compatible?

    florent: we would want to confirm that indeed
    … sending too much data closes the data channel, so developers
    already need to pay attention

    Bernard: the only time you would see this is in some weird
    maintenance scenarios - it should be very rare

    florent: we can add some measurement in Chrome to see if that

    dom: +1 to these solutions if they're web compatible

    florent: so we can start with copying the value in opening, and
    measure web-compatibility of rejecting a renegotiated size

    jib: would maxMessageSize end up being exposed on the data

    florent: we could do that, but that's not part of this proposal
    … this wasn't useful in the context of running everything in
    the same context as peerconnection
    … but with transferred channels, this makes more sense to

    dom: it would be clunky not to expose it

     Issue [88]#115: Need to specify behavior of detached RTCDataChannel
     objects [89]🎞︎

      [88] https://github.com/w3c/webrtc-extensions/issues/115
      [89] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=4276

    [90][Slide 34]


    florent: we need to document a [[Detached]] internal slot per
    the HTML spec for transferable platform objects
    … we would keep [[isTransferable]] for the a datachannel that
    has already sent

    [no objection]

    jib: it remains unclear what happens to data channels when
    they're transfered in the main thread

    [91][Slide 35]


    florent: should transfered data channels be garbage collectable
    in the main thread? they're "closed" which make them
    collectable without a strong reference
    … we could add a new state "detached" on top of opening, open,
    closed etc

    Bernard: I prefer Proposal 2

    jib: transferable objects are more like a clone, leaving an
    unoperative a clone
    … so the broader question is how a [[Detached]] data channel
    should behave, how it should affects the existing algorithms

    florent: because they're closed, this already impacts the
    methods close() and send()

    florent: hearing some support to introducing a "detached"
    state, and a  [[Detached]] internal slot

    hta: what about garbage collection?

    florent: let's discuss on github

   [92]Capture Handle [93]🎞︎

      [92] https://github.com/w3c/mediacapture-handle
      [93] https://www.youtube.com/watch?v=cbLpKnU6RoI#t=4738

    [94][Slide 39]


    [95][Slide 40]


    [96][Slide 41]


    [97][Slide 42]


    [98][Slide 43]


    [99][Slide 44]


    [100][Slide 45]


    [101][Slide 46]


    [102][Slide 47]


    [103][Slide 48]


    [104][Slide 49]


    Elad: the proposal is to add some structure to capture handle
    … fo crop targets (possibly with specific content hints)

    jib: what about a messageport?

    elad: still not structured, so leads to tight coupling

    jib: I'm not sure we want to specific all the different things
    that application might need to agree
    … per [105]#11, I don't think we should re-invent postMessage

     [105] https://github.com/w3c/mediacapture-handle/issues/11

    elad: a messageport informs the capturee they're being captured
    … capture handle is a unidirectional message port
    … being able to update the handle is useful given that the
    captured content is going to change
    … a messageport can be useful in general, but for different use

    jib: can we take a step back to understand the requirements we
    … what API surface would be expose here?

    elad: adding structure for a crop target in the capture handle
    instead of a simple string
    … croptarget would have contenthints, and also add a
    messageport as a separate suggestion

    dom: I think maybe a unidirectional messageport would work for
    what we want?

    elad: several suggestions: move from string to object in
    capture handle - already needed for tightly coupled apps
    … for loosely coupled apps, similar to what capture actions
    already allow, adding explicit support for croptargets /
    contenthints would go a long way to help

    elad: what about the first suggestion - moving from a string to
    an object?

    jib: would re-iterate [106]#11 - let's not reinvent postMessage

     [106] https://github.com/w3c/mediacapture-handle/issues/11

    elad: but this adds ability to decouple capturees/capturer

    jib: but adding this to the browser API when it facts it's down
    to the app to use it or not
    … that's odd

    elad: it's similar to capture actions, not really more
    formalized in semantics

    hta: are there establishing standardized protocols over
    messageport already?

    dom: don't know off the top of my head, would have to check

    hta: if we were to have to come up with that, this feels scary

    dom: re going with an objects, would that be for serializable

    elad: yes

    jib: the original purpose for handle was an identifier; now
    we're talking about passing objects, that changes the nature of
    the API

    [107][Slide 51]


    elad: a messageport doesn't address all the use cases - it's
    not structured
    … I'm hearing support for the use cases, and not seeing an
    alternative proposal

    jib: I remain a bit lost on the requirements we're solving with
    this API
    … e.g. it could be a separate field instead of being part of
    the handle
    … I'm not sure why should allow random web sites to specific
    crop targets

    elad: slide 47 illustrates how this could be a purely
    user-driven process to avoid any user tricking

    jib: but I'm not sold we need to allow this for random web

    harald: what criteria would a web site eligible to this?

    jib: with a messageport?

    elad: but that makes it more likely to create situations where
    a web site might want to trick another provider?

    jib: I still don't see a compelling case for making handle an

    elad: is the video provider / vc collaboration use case

    jib: yes - we should figure a better way

    elad: what way though?

    hta: I'm hearing 2 proposals: make handle with some pre-defined
    fields for specific purposes (e.g. listing croptargets); and a
    messageport for tightly coupled apps
    … these are 2 independent proposals that should be evaluated

    jib: would allow any serializable object be safe to expose to
    the capturer? that seems problematic

    elad: the security properties are similar (or even somewhat
    safer) than a messageport

    Ben: with arbitrary objects, could that raise OOM concerns?

    elad: 1. The captured page would be attacking itself first and
    … 2. The captured page would be attacking an unknown capturer
    that likely doesn't even exist.
    … 3. We can neuter the attack by ensuring the capture-handle is
    no-op on the capturer if the capturer does not read the handle.
    But that's for the future, if the attack comes up in the wild,
    which is unlikely.

    ben: are there objects that could create risks for the

    elad: not that I'm aware

    dom: I think the chairs will have to propose steps to unblock
    this conversation
    … maybe an explainer would help figure out all the
    considerations that need to be taken into account

    hta: the chairs will do so

Summary of resolutions

     1. [108]pass an array arguments to generateKeyframes
     2. [109]Add mimeType to metadata
     3. [110]move forward with framesCaptured only for now
     4. [111]proceed with the proposed clarification
     5. [112]close [113]#2762 as is

     [113] https://github.com/w3c/mediacapture-handle/issues/2762
Received on Tuesday, 25 October 2022 17:17:36 UTC

This archive was generated by hypermail 2.4.0 : Tuesday, 25 October 2022 17:17:37 UTC