- From: Dominique Hazael-Massieux <dom@w3.org>
- Date: Wed, 20 Nov 2024 15:35:47 +0100
- To: "public-webrtc@w3.org" <public-webrtc@w3.org>
Hi,
The minutes of our WebRTC WG meeting held yesterday are available at:
https://www.w3.org/2024/11/19-webrtc-minutes.html
and copied as text below.
Dom
[1]W3C
[1] https://www.w3.org/
– DRAFT –
WebRTC November 19 2024 meeting
19 November 2024
[2]Agenda. [3]IRC log.
[2] https://www.w3.org/2011/04/webrtc/wiki/November_19_2024
[3] https://www.w3.org/2024/11/19-webrtc-irc
Attendees
Present
Bernard, Carine, Dom, Florent, Guido, Harald, Henrik,
PatrickRockhill, TimP, Youenn
Regrets
-
Chair
Bernard, Guido, Jan-Ivar
Scribe
dom
Contents
1. [4]Timing Control in MediaStreamTrack Insertable Media
Processing using Streams
1. [5]Issue #80: Expectations/Requirements for VideoFrame
and AudioData timestamps
2. [6]Issue #114: VideoTrackGenerator/MediaStream should
buffer the current frame
3. [7]Issue #20: Add “real-time” warning/note to
`MediaStreamTrackGenerator`
4. [8]Issue #86: Playback and sync of tracks created by
`VideoTrackGenerator`
2. [9]WebRTC API
1. [10]Issue 229: receiver.hardwareAcceleration attribute
instead of disableHardware[En|De]coding() static
methods Issue 229: receiver.hardwareAcceleration
attribute instead of disableHardware[En|De]coding()
static methods
2. [11]Issue #3017: Reconsider replaceTrack's blocking on
the operations queue
3. [12]Issue #3022: Clarify order of events when the SCTP
transport is closed
4. [13]Add event for CSRC change
5. [14]Issue #3020: Codec negotiation and level IDs
6. [15]Remote channelCount + Stereo Opus #3010 #3011
3. [16]Summary of resolutions
Meeting minutes
Slideset: [17]https://docs.google.com/presentation/d/
1qFWqfbOVv3mHlYCBtnmXtLdpMkk-pXf--cGoEQBCyi4/ (
[17]
https://docs.google.com/presentation/d/1qFWqfbOVv3mHlYCBtnmXtLdpMkk-pXf--cGoEQBCyi4/
Timing Control in MediaStreamTrack Insertable Media Processing using Streams
Issue #80: Expectations/Requirements for VideoFrame and AudioData timestamps
Jan-Ivar: what we want to do is to write tests to see how browsers
proceed today, and ideally prescribe that
Bernard: but where? in HTML video element?
HTA: I think it'd belong there; what do video elements do when they get
the same timestamps?
Youenn: each sink is supposed to define how it handles videoframes
… In Safari, we use the time at which the frame at which it is committed
the sink
… webrtc-pc should define what it does when sending frames
… the HTML video element should define it, although we could do in
mediacapture-main if it's easier
Bernard: that's a good point - the track could go to a peerconnection
where ti would be weird to have these duplicate timestamps
Youenn: we need to test, but I suspect they're using the time at which
frames are submitted
Guido: +1 to youenn
… this would also mean specifying for mediatrackrecorder,
mediatrackprocessor
youenn: let's make sure to file issues for all these specs
PROPOSED: Add to mediacapturem-main extensiblity to make sure sink
define their behavior on frame timestamps and file issues on sink specs
accordingly
TimP: if you were to feed a track with null timestamps to a video
element, would it be different if it had gone through a null transform?
that would be weird
hta: a null transform would do nothing
TimP: right; but an identity transform shouldn't change how the track
gets rendered
dom: as long as we're defining at the sink, this means the processing
pipeline shouldn't change the outcome
youenn: it may be that we can define a default behavior for sinks
(likely, changing the timestamps would have no effect)
bernard: that's my hope too; PC is an interesting case
Jan-Ivar: in the case of a transform, a Processor WOULD be a sink - so
we should be careful we keep the invariant of identity
RESOLUTION: Add to mediacapture-main extensiblity consideration to make
sure sink define their behavior on frame timestamps and file issues on
sink specs accordingly
Issue #114: VideoTrackGenerator/MediaStream should buffer the current frame
Youenn: +1 on not buffering; I'm surprised that captureStream() is different
… we should check if there isn't a misunderstanding on the behavior of
captureStream()
Jan-Ivar: it would be nice if we could keep them consistent, although I
understand the cost of buffering
Youenn: but I don't think it's even buffering, it's only a timing question
Henrik: clearly we shouldn't buffering in the traditional sense; but in
terms of remembering the last frame, in the case of a variable framerate
track, is delivery guaranteed?
Bernard: that may deserve another issue
Henrik: +1 on this being the expected behavior
HTA: the <canvas> element exists no matter, so a frame can be generated
at any time; I think this is expected behavior. Seeing if we can align
the polyfill with this, that would be best and informative
RESOLUTION: Not buffering the last frame is the expected behavior
Guido: we can recommend to the developer that they can buffer the frame
in the application and send it to the generator after the track is
associated to the element
Youenn: note that this problem only occurs when using VTG in the main thread
Issue #20: Add “real-time” warning/note to `MediaStreamTrackGenerator`
Youenn: with VTG's writable stream, the promise could be awaited as a
cue to when a new frame can be submitted
… not sure if it's needed or still web-compatible
Bernard: what happens if you oversubmit? is it only they're not being
displayed by the sink?
HTA: I think they would be thrown away
Jan-Ivar: a writable stream normally comes with a queue with a highwater
mark; how should the VTG infers its framerate from the pace of frames it
is being fed? Maybe we should define a max framerate over which it gives
backpressure
… or maybe it just drop frames over that rate if we're sticking to
real-time behavior
… the stream can either be push- or pull-based; for pull-, we need to
determine where the framerate gets identified
youenn: the write-frame algorithm returns a promise, which resolves the
writable write(), which gives a path for backpressure
… at the moment, the promise always resolve at the end of the microtask,
but we could change it to allow for backpressure
… at the high level, I don't think VTG should be lossy, while sinks might be
Guido: +1 to youenn
… if you don't wait for the promise, you have no guarantee that the
frame will be processed
henrik: the promise helps with avoiding pushing too much data
… but for lossy sinks, there will still be no way to make sure the frame
got processed
dom: ideally, lossy sinks would be able to transfer backpressure via VTG
youenn: this would be possible for PC; possible for recorder; not sure
about media elements, although in case of too many frames they should be
able to at least render the last
… I think the invariant is that VTG is not lossy, but some sinks will be
lossy, and we should say that the last frame should win for renderers
jan-ivar: a writable always has a queue, even if the sink doesn't have one
… we need to understand how the framerate gets determined for a
pull-based stream
Bernard: this will depend on pre-recorded vs live
Issue #86: Playback and sync of tracks created by `VideoTrackGenerator`
HTA: looking at the reported bug, the person reporting it had specific
ideas on how the video element work, e.g. with playout points
Jan-Ivar: I don't know the answer, but writing tests would help
determine if timestamp has any effect, in which case browsers are likely
optimizing local playback
Youenn: Tracks are sent synchronized by the PC, send asked to render
ASAP given the real-time context; likewise for local capture; I don't
thnk timestamps are playing any role in it
Henrik: I thought presentationTimestamp was being used in Chrome?
Jan-Ivar: that should be testable
WebRTC API
Issue 229: receiver.hardwareAcceleration attribute instead of
disableHardware[En|De]coding() static methods Issue 229:
receiver.hardwareAcceleration attribute instead of
disableHardware[En|De]coding() static methods
Youenn: not sure whether it's worth exposing just that; there are
proposals to expose webcodecs data to the PC
… I'm not sure if the implementation will be able to determine if
switching to software or hardware is adapted given impact on latency,
drops given the opacity of the PC flow; this is much different from the
WebCodecs context where everything is happening under the control of the JS
… I would prefer we spend more time on exposing WebCodecs rather than
providing this hard-to-exploit toggle to developers
… I'm not more in favor of the new API than the current one
Bernard: the original motivation was to turn-off hardware acceleration
to avoid encoder bugs
… this new approach gives control which as Youenn describes wouldn't be
operational
Henrik: I think this new proposal is best because it allows to have the
remote party signal to the specific client they should disable their
hardware encoder
HTA: I recall as Bernard that this was a toggle to remove hardware
encoder on detected failure pattern
… this new proposal is more flexible; the "preferSoftware" might be too
soft for the original purpose though
Jan-Ivar: one of the reason of making it softer is to allow UA to
override the hint when e.g. they've been blocklisted due to issues in
earlier versions of the browser
HTA: hearing two proposals: Youenn suggest deleting the old API
altogether; the other is to add this new API
Bernard: with encoded sources, we're heading towards WebCodec control
which will give you the granular control if you need it
Youenn: +1 to Bernard
… encoded sources bring that to the encoder side; we don't have the
equivalent on decoding yet, but would be a reasonable direction to explore
Henrik: how about my suggestion to rely on the remote corruption metrics?
Youenn: you could hot-download a WASM decoder when you detect that situation
Jan-Ivar: not hearing a lot of consensus on the proposal; I expect
Mozilla's position will return to negative to the existing API
Repository: w3c/webrtc-pc
Issue #3017: Reconsider replaceTrack's blocking on the operations queue
HTA: +1 after putting more thoughts on this
Youenn: sounds fine, and helps getting a bit faster with limited compat
issue
[Henrik: +1 on reaction]
RESOLUTION: move with forward with PR to align spec with Chrome behavior
Issue #3022: Clarify order of events when the SCTP transport is closed
Youenn: I think we should enqueue a task per event; in terms of order
for data channels, I'm not sure if there an API with an order - there
used to be a getDataChannels() ordering which we could consider reusing
… re closed transports, my intuition we should fire on the SCTP
transport first, but no concrete reasoning
Jan-Ivar: we could use the orders of the internal slot, although not
sure how that intersects with datachannels that are transfered
Florent: the sctpReason property gives indication about what caused the
closure of the channel
TimP: I could live with getting notifications only at the transport
level when the full association goes down
Jan-Ivar: if we fire first on SCTP, then you would know why all the
channels close subsequently
TimP: right; in particular, you don't want to just re-open the channels
if the transport itself has failed
Youenn: I think it's useful to still emit the close events on channels
in particular when they've been transfered to a worker
HTA: note that there is also an error event fired
RESOLUTION: Proceed with PR to fire sctp closed on transport object and
then on individual channels
Add event for CSRC change
Bernard: the original use case wasn't to have extensive polling; there
was also no expectations to manage as large meetings as is the case today
… I think the main use case is for audio rather than video to surface
where sound is coming from
… there was concerned about the volume of events that could be fired
from this as well
Henrik: this would be only when the source changes, so it shouldn't a lot
Jan-Ivar: my concern is the relationship to media flowing, and the
linkage to the mute event which is already being fired
Henrik: I asked and they indicated they're interested in source information
Jan-Ivar: but I'm concerned in introducing an API that would cement the
interop gap we have today
Jan-Ivar: this could be expressed in the standard's position; I'm trying
to make sure we distinguish implementation issues from use case discussion
Youenn: the idea is that when a CSRC or SSRC change is detected, a task
would be queued to fire an event with the changed value?
… if so, seems OK with me
Jan-Ivar: I'm not sure if the csrc/ssrc names are evocative enough for
the use case
Henrik: this matches the existing names of the getters
Bernard: for the video, would this be coming from the video compositor?
or if the mixer mixes audio from different sources?
Henrik: it would probably send an array with the various sources?
HTA: we know there are apps that do this today via polling; what we're
looking for is exposing this in a more ergonomic and efficient way
Jan-Ivar: re the use case - polling isn't necessarily bad
Henrik: you would still need polling e.g. to animate an audio volume
… but having information asap on a change to e.g. switch the main video
is time critical
… we've received several reports where this polling is creating glitches
and impacts performance
Bernard: it's kind of a new use case from the original one that was
focused on audio - but now this also extends to video, is that common now?
Henrik: I know at least of a Google product doing this
Youenn: this aligns with usual Web expectations in terms of signaling
changes to states
Jan-Ivar: my support is tentative, given my concern about the lack of
interop of mute event
Henrik: we can make that clear in our intent to ship for Chromium
Youenn: implementing this new API could help improve implementations of
the mute event; for Safari, we would like to align with spec and I think
this would help
Henrik: fixing onunmute along with implementing this makes a lot of
sense to me
… onunmute makes sense regardless since not all apps used virtual ssrcs
… the lack of support has mostly been a question of prioritization
Guido: a lot of the mute issues were linked to the ambiguity in the
mediacapture spec that crept in multiple specs; this is something we
want to fix
… see e.g. the proposal I made to expose additional stats to help with
transition
Jan-Ivar: onunmute also useful as trigger for gathering stats
RESOLUTION: Rough consensus on moving forward with events for csrc/ssrc
changes, with concerns noted on related onunmute interop
Repository: w3c/webrtc-pc
Issue #3020: Codec negotiation and level IDs
Bernard: you said before negotiation, 180 throws because you have no
codec; this relates to what constitutes the media format
… for H265, it's profile-id and tier only, so setting level-id to 180
should not throw
Henrik: I think the current spec says "equals" which could be clarified
… but even with that, there is still another issue that my next slide
details
Jan-Ivar: I think it's OK for the code to fail before negotiation
Henrik: you're in proposal B camp then :)
Henrik: I personally prefer proposal A :)
Bernard: me too
Jan-Ivar: A makes sense after hearing your argument that it could
produce unexpected results in prod
HTA: proposal A, and we should document that when you can't negotiate
the codec, it should throw
RESOLUTION: Proposal A
Repository: w3c/webrtc-pc
[Youenn departs]
Remote channelCount + Stereo Opus #3010 #3011
Jan-Ivar: re how to test it, with Web Audio
Henrik: still I think it would better to add it for symetry
Jan-Ivar: adding it seems OK to me
RESOLUTION: proceed with adding .channelCount on remote tracks
TimP: I'm almost sure that the SDP for Opus is always stereo
Henrik: I think the spec says it's always stereo capable; but the SDP
parameter only expresses a preference
TimP: but Opus on the wire I think will always be stereo
Henrik: I think it can be mono on the wire (since I understood it can
impact bitrate)
Jan-Ivar: it's hard to detect a mono signal from a stereo track; having
that exposed in channelCount would be useful
… re channelCount, would this trigger onconfigurationchange?
Henrik: I haven't thought about it
HTA: I talked with Jean-Marc about mono & stereo; if you know a signal
is mono you can encode/decode with less CPU power, which is good
… re testing, generate a track with different signals (e.g. a waveform
on one channel and silence on the other) and see whether that's preserved
Jan-Ivar: re proposal 2, "MAY" is not great
Henrik: I suspect implementations would align with what libwebrtc does
Jan-Ivar: let's confirm it and standardize that
… if the libwebrtc behavior isn't satisfactory, we can revisit that
Received on Wednesday, 20 November 2024 14:35:49 UTC