- From: Francois Daoust <fd@w3.org>
- Date: Wed, 09 Jun 2021 09:25:16 +0000
- To: public-media-wg@w3.org
Hi,
The minutes of yesterday's Media WG call are available at:
https://www.w3.org/2021/06/08-mediawg-minutes.html
... and copied as raw text below.
Please note that a follow-up call on WebCodecs issues is planned on
Tuesday next week at 3pm UTC:
https://lists.w3.org/Archives/Public/public-media-wg/2021Jun/0001.html
Thanks,
Francois.
-----
Media WG monthly call
08 June 2021
[2]Agenda. [3]IRC log.
[2]
https://github.com/w3c/media-wg/blob/main/meetings/2021-06-08-Media_Working_Group_Teleconference-agenda.md#agenda
[3] https://www.w3.org/2021/06/08-mediawg-irc
Attendees
Present
Chris Cunningham, Chris Needham, Dale Curtis, Dan
Sanders, Eric Carlson, Eugene, Francois Daoust, Gary
Katsevman, Jer Noble, Lei Zhai, Mark Watson, Nigel
Megitt, Paul Adenot, Peng Liu, Tess O'Connor, Youenn
Fablet, Zachary Cava
Chair
Chris Needham, Jer Noble
Scribe
Chris Needham, Francois Daoust
Contents
1. [4]Rechartering update
2. [5]TPAC 2021 planning
3. [6]Welcome Alastor Wu as editor for AutoPlay Policy
Detection
4. [7]Codec switching in MSE & FPWD
5. [8]WebCodecs issues
1. [9]Should WebCodecs be exposed in Window environments?
Meeting minutes
Rechartering update
tidoust: AC review ongoing. Please review the draft charter and
vote on it. The more support the better.
TPAC 2021 planning
cpn: TPAC is W3C's annual "F2F" meetings. Proposal is for TPAC
to be organized over 2 weeks: one week for joint meetings, one
week for breakout sessions.
… One planned joint meeting with WebRTC WG, e.g. to discuss
WebCodecs.
… I suggest to focus TPAC on joint meetings.
Tess: If the group can do 1-2 day during that week, that's fine
with me. I would recommend against scheduling a group meeting
one week before that otherwise TPAC spans too many weeks in a
row.
chcunningham: Happy to focus on joint meeting given the virtual
nature of the meeting.
cpn: I can say that the Media & Entertainment IG will want to
organize a joint meeting. That's me with another chair hat on.
… Open for suggestions otherwise!
jernoble: I just want to make sure that everyone gets a chance
to be heard. If you want to meet, please say so!
Welcome Alastor Wu as editor for AutoPlay Policy Detection
jernoble: Alastor is not around today, but thank you for
volunteering for this job! Looking forward to discussing
Autoplay Policy Detection.
cpn: Alastor [10]raised some questions on the spec. We'll need
feedback from other implementers.
[10] https://github.com/w3c/autoplay/issues/12
jernoble: Looking into it.
chcunningham: We'll discuss it internally and get back to you.
dale: You can use me as a contact in the meantime.
Codec switching in MSE & FPWD
tidoust: The [11]pending pull request means we've added the
main feature the group is set to work on for MSE. Can we issue
a CfC to publish FPWD once the PR lands?
[11] https://github.com/w3c/media-source/pull/274
jernoble: what's the plan for MSE on the Rec track?
tidoust: If there are additional features you think should be
in the FPWD, we can wait. FPWD starts a call for exclusions, so
it's a good idea to have the features you're planning to have
in there. It does not have to be perfect.
jernoble: MSE in workers has already landed in Chrome. Also
playback through gaps in unbuffered ranges. That work has
already been done. If it's ready to go, we can start the
publication process
tidoust: In that case I'd suggest postponing until those
features are specified
dale: Matt's timeline is around June 30th, should have more to
say next meeting. We're working towards an origin trial for MSE
in workers
jernoble: A good opportunity to tie it all together for the
FPWD
WebCodecs issues
jernoble: 3 outstanding issues on WebCodecs
… Design questions. Want to make sure that everyone gets heard
and see whether we can come to consensus.
Should WebCodecs be exposed in Window environments?
[12]Should WebCodecs be exposed in Window environments? (#211)
[12] https://github.com/w3c/webcodecs/issues/211
chcunningham: WebCodecs are currently exposed on Window and
workers
… WebCodecs does not do heavy lifting on the main thread.
… It was intuitive to us that this got exposed to window.
… The points raised by Youenn are about performance concerns:
if main thread is with lots of contention, it becomes
problematic to queue and manage your inputs and outputs.
… We think that's why workers are a good fit for many
applications.
… That's not the case for all applications though.
… There are applications that are not latency-sensitive or that
have to deal with main thread contention.
… During last call, it was suggested that we polled out origin
trial users.
… Many of them didn't have an opinion. Those who expressed
opinions favored exposure to window.
… They reported on applications without performance
constraints.
… Additional points: workers add complexity. Also integration
with APIs that are not currently exposed to workers. This may
be the case in the future but it is a pain for developers in
the meantime to integrate with these APIs.
… We did [13]hear from the TAG yesterday that, following the
dev survey, they found that it was good to expose to window
after all.
[13]
https://github.com/w3ctag/design-reviews/issues/612#issuecomment-855754952
Youenn: My perspective is that it is more on what we should
prioritize.
… FWIW, we had a very similar discussion in the WebRTC WG about
MediaStreamTrack.
… It's not finished. My understanding is that we are leaning
towards "worker support is great, let's do that first. Window
support is second target. We want to focus on worker and
optimize for that."
… With WebCodecs, I'm mostly focusing on video codecs. It's
very similar to MediaStreamTrack.
… We know that we won't regret exposing to workers.
… We know that, initially, some people will have to create a JS
shim with postMessage and the like, which is a bit sad, but it
will push other APIs to workers down the line.
… Moving more stuff out of the main thread is a good thing.
… Some people are saying that there is no benefit so don't want
to pay for the added complexity.
… That's not entirely true. There will be repetitive tasks in
the main thread. There will be GC calls. If you push to
workers, you can better handler user events in the main thread.
… UAs can also decide to lower the priority of transcoding
workers that run in the background. More difficult to do if
transcoding is running in the main thread.
… In other words, end users may benefit from the additional
work.
… That's why I believe that we should push them towards that.
We may not want to mandate that in the end.
… Avoid situations where devs say that "it works on my machine"
and believe that it's going to work everywhere.
… Start with things we won't regret.
… [WebRTC example]
… This is the world I want people to live in, and the reason
I'm pushing for exposing WebCodecs to workers
… Media capture track processing in web??.io. We got that
feedback from an xCloud dev: "all your examples should be in
workers". Main thread demos are going to be copied and pasted,
leading to bad design. If you prototype things in window, you
may remain in window, that's my fear.
chcunningham: Youenn wants to wait to see if this flies. My
perspective is that the trial already demonstrates that, as
evidenced by feedback received. They found no significant
thread contention.
… They tried workers as well and found no real improvement.
… Example of game engine that does decode and render.
Performance was good as well, and worried they might have to
switch to workers.
… Did not quite follow the point on GC.
… Codec work could be de-prioritized too if main thread is
contended.
… Agree that the spec or doc should highlight API use in
workers
Youenn: Re. de-prioritizing, you will need to keep the queue,
so that's difficult. It could be done with changes.
… I think we can quickly find consensus and move forward with
workers.
… Less sure about exposition to window.
… Given where we're at with browsers stacks today, some things
that could improve the situation are not there yet.
… Re. GC, easier to optimize in a worker because you fully
control the script. In the main thread, you might not own the
entire JS, so more difficult to control when the GC is going to
kick in.
… [Mentions MediaRecorder as a way to do encoding in the main
thread for people who do not want to use workers]
… We already have some APIs for decoding audio on the main
thread. Not perfect for sure. Instead of adding a new API, we
could improve them.
… WebCodecs is going to be successful in low-latency realtime
use cases. It may shine in other use cases, but we should get
it right in workers first.
Eugene: About low-latency use cases, as I see it, having
WebCodecs only on workers increases latency. It creates
extra-hops between window and workers. Our colleagues from
Stadia were concerned that input APIs are not available in
workers, so they'd need to inject extra-jumps.
Youenn: Google Stadia uses WebRTC right now, I believe. Not at
a frame-by-frame level. Why the move to WebCodecs?
… You have data channel in the worker. You pipe it to
WebCodecs, then you get a frame, then piping into rendering.
… If you need to go through network or something else, we're
missing something.
Eugene: WebCodecs being a lower level, Google Stadia is
exploring and prototyping.
… I don't want to go into details for them, but that's kind of
patronizing for us to assert right ways to do things for them
and others.
… Different designs could be working for them.
… It's not only about decoding and outputing stuff.
jernoble: It seems like progress could be made by having
specific examples about pages that use WebCodecs, to promote
move to workers.
… There is specification experience in doing things in workers
not to contend the main thread, typically for audio rendering.
… We used to have bits in main thread, and had to backtrack as
it was found to be bad design.
chcunningham: WebCodecs exposes decoded audio, but main use
case will create a jitter buffer, etc.
jernoble: Re. "patronizing", we're trying to get the best
end-user experiences. I don't believe that it is us
patronizing.
… We're not intending to patronize people and tell them how to
do their job. We're trying to create best UX for end users.
… We may be able to help with complexity by writing
demonstration cases that show how to do things in workers.
… If we are intending to use this API on the window, one thing
that this API could improve is providing queueing hooks.
… so that you can propose multiple frames to the decoder so
that it always has something to chew on.
chcunningham: I don't follow the "queueing makes GC harder"
argument. The queue is an internal detail at that point.
… Re. GC on window not owning the full script, I feel we're
talking past each other. The use cases I have in mind own the
JS, they know exactly all the bytes that are going on there.
… No objection to extending MediaRecorder. But extending it to
do things that WebCodecs would do if it were exposed to window
seems a bit strange.
… WebCodecs does not have the same requirements in terms of
timing for audio.
… Re. priority of constituencies, I get that but sites also
consider themselves as agents for end users and most are doing
a good job at making sure performance and UX are good.
… TAG was sympathetic to Youenn's points initially, but now
we've heard from the users, and TAG is favouring window
exposure.
… That speaks to my previous points: we've heard that worker is
not enough.
jernoble: Sites don't have the ability to control whatever
happens in the main thread. Other pages may be running on the
same process.
… The user is the person using the computer. It is our job to
be opiniated about this.
… I don't think using the terms you and Eugene have been using
is useful.
cpn: What do we do next? It seems some higher-bandwith
conversation before next monthly call would be useful.
hober: There are strong opinions. I'm sad I wasn't in the TAG
breakout where this got discussed.
… Re. "objections have been considered", in a WG like this that
runs on consensus, we need to establish consensus first before
we move on.
chcunningham: I agree that we do not have consensus for now.
… I would like to suggest that we issue a call for consensus on
this issue. The points raised today were mostly a summary of
the points raised in GitHub. I feel that we're going in circle.
… We have heard from developers. We have heard from the TAG.
… Would it be possible to issue a call for consensus on this?
cpn: My understanding is that a CfC will bring forth statements
of positions. We'll establish what we already know that there
is not yet a consensus.
… The problem is how do we resolve that?
… The W3C process talks about following the path that creates
the least dissension. Perhaps that could be the path we follow.
… If we expose the API to both window and workers, devs will be
guided by examples. If they are good, that could be enough.
… It seems to me that there is another WebRTC angle to this
that we could perhaps discuss as this seems to originate from
there.
Youenn: Yes, it is true that it follows from another WebRTC
issue.
chcunningham: Re. CfC, I don't expect it to lead to consensus.
The process would do two things: it would make positions
explicit, and it forces the chairs to break a tie.
jernoble: I don't think that it is the role of the Chairs to
break a tie. We're here to encourage consensus.
jernoble: This conversation is a bit narrow, others should
weigh in.
… It may be useful to find other solutions to concerns raised
here, such as good practices, examples, other API features that
could solve them without removing the API from window.
cpn: I would forward putting forward a proposed resolution
along these lines: with supporting documentation, we would move
forward with window and worker. We'd be guiding towards most
appropriate usage.
… Also supportive of running another meeting. Two other issues
to discuss.
chcunningham: I personally would be fine with that proposal.
Happy to have another meeting.
jernoble: OK, cpn and I, let's figure that out ASAP.
… Let's document additional points that we discussed today in
the GitHub API.
… Apologies for running overtime!
… Hope to see everyone in the next conference
Minutes manually created (not a transcript), formatted by
[14]scribe.perl version 136 (Thu May 27 13:50:24 2021 UTC).
[14] https://w3c.github.io/scribe2/scribedoc.html
Received on Wednesday, 9 June 2021 09:25:37 UTC