- From: François Daoust <fd@w3.org>
- Date: Thu, 06 Apr 2023 13:53:03 +0000
- To: public-webtiming@w3.org
Hi, Notes from yesterday's Multi-Device Web Timing CG call are available in the following doc: https://docs.google.com/document/d/1NQjX5uQ4fFma854yQml0JxkZ74mzwEe-mvvAOJfbGC8 ... and copied as raw text below. We discussed several leads to progress the Timing Object proposal, including: - The creation an ACID-like test, or help develop test suites. - Renaming the group to drop "multi-device" to clarify that this is also useful for single device scenarios. - Developing a media player prototype using WebCodecs to showcase how synchronization could work. - Re-aligning the spec with current implementations. All of these require committing time, which call participants cannot easily do for now. If they materialize, plans to support synchronization primitives in existing media player libraries could perhaps provide some incentive to pursue the work. Thanks, Francois. ----- Minutes of Multi-Device Timing CG calls 5 April 2023 Present: Ingar, Njaal, François, Chris, Christoph Agenda: https://www.w3.org/events/meetings/1357a24a-f8d0-4de9-a33e-b14ece5bf677 Scribes: Chris, François Progressing the Timing Object spec Francois: Main question for me is what do we want to do with the spec? Christoph: Can it still be a W3C spec even without browser implementations? Francois: I don’t see any reason W3C can’t be the home for the spec, what matters is the community around the spec. I note that Apple used some of the ideas in the spec. W3C is usually a place for browser specs, but this is a community group, so there’s no requirement it be a browser spec, and no problem as it’s still web stuff. C.f. Media WG discussion where Apple introduced Media Session Coordinator Ingar: It was a good thing to do in W3C, gave it some recognition. Francois: CGs are good for work that’s close to standardization. Ingar: Back in the days, we wrote the proposal thinking that browsers would implement it. But we quickly realized that this might not be necessary. No real need for browsers support in the end, except some changes in media frameworks to facilitate syncing. Pushing browser vendors to make necessary adjustments and perhaps improve their test suites are also valuable goals. Njaal: If browser vendors wanted to fix media sync, it would be weird to base that on a user lock. We reported a bug against Safari about 12 years ago. Accepted as a bug but still no fix. Chrome also tends to lie on the time when you play with playbackRate. Christop: Yes, there are also bugs in Firefox as well. I have the feeling that, at large, no one is really interested in it. There’s a niche for it, for sure. Njaal: For TV consumption, two extremes: one or two seconds out of sync is fine, or 0.1s is needed. Our code hasn’t been updated in the past 5-6 years. I have started to refresh it. I think that we’ll see some new issues, e.g. with iOS and multiple videos. Main difficulty is maintaining code without dedicated resources. It still works surprisingly well though! For most use cases, no problem, but we’ve seen reports about drifting after some period of time, e.g. 30mn in MP3 playback. Main issue is that this is not being tested in browsers. It is of course possible to make a web page that could create a “score” for it. Ingar: Details about the behavior of media players for instance when you fiddle with playbackRate, I don’t know whether that’s well defined. Njaal: You can easily measure things: latency, variance, etc. It does not necessarily translate to whether it’s controllable. That could be useful but we also need browser vendors to actually use it. ACID test wouldn’t be fun if no one used it. Ingar: You would also want to integrate Web Audio as that too is often used for synchronization. Christoph: One thing that would be difficult to integrate is that measures need to come from out of band, as you cannot rely on the browser to give you the right measures. Chris: In some of the internal tests we’ve run, we had a camera pointing at the screen for that purpose. Difficulty is that most of it is a “quality of implementation” issue. One of the libraries that I have is an audio visualizer. I occasionally receive feedback that it is not perfectly in sync. It goes beyond multi-device. Single-device sync is also important. Ingar: In that sense, “Multi-Device” is a misnomer. More “multi-element” synchronization. Njaal: I often create pages with half of a video on one side and the other half on the other side. There, you immediately see when things are out of sync. That’s how I see e.g. problems with Chrome reporting a wrong currentTime until you do a little skip. Christoph: No way to know it from within the browser itself, right? Njaal: A bit complicated. We could pause and copy frames to a canvas and compare things, but that’s heavy. For audio, it’s even worse. We did tests in the past where reported numbers were fantastic and yet audio was out of sync, which you notice if you plug one device into your left ear and the other into your right ear. These things are a lot more difficult to validate automatically. Chris: In that case where currentTime goes inaccurate, is there a Chromium bug that describes that? Njaal: I don’t think so. Christoph: It’s hard to reproduce in practice. Or rather, complicated, involving playing the media for a few minutes. One thing that I wanted to report is that, in Chrome, when playbackRate is almost 1, they round it, you will accumulate drifts. Range needs to be more 0.9 and 1.1 instead of 0.999 and 1.001. Wouldn’t it be possible to create a minimal media element implemented through WebCodecs and OffscreenCanvas? Chris: I’d love to see that. If you do that, you’re taking the sync control out of the browser. But then you have to do all of the buffering yourself, which is much more complicated. Njaal: I wouldn’t be surprised if there were other delays that cannot be accounted for easily, such as hardware delays. Chris: One of the open questions we have on WebCodecs is whether to expose whether hardware is used. If software is used, timing can be further tightened probably. Njaal: I remember using GStreamer in the past. The good thing is that I could prepare the pipeline and at the time when you hit play, it plays immediately. In the browser, play will start with some lag. For me, synchronizing would be so much easier if the media element had a pre-roll function. You don’t only want to preload the data, you want the pipeline to be ready. I was hoping that the “pause” function would do that. It does not. If TimingObject was used, there would be no problem. Ingar: The ease of use of TimingObject is probably more important than its precise sync abilities which are only appealing to a niche. One of the use cases that we’ve heard about is: Live feed from a lottery and also the results. If the results appear before the video shows the results, that’s not good. People point at low-latency video streaming for that, but that’s complicated and perhaps not that needed. Things could be easy but are not because of the lack of sync primitives. Point is that interest could be much larger than the presumably small group of people who care about precise sync. Njaal: Good use case with multi-camera perspectives. Now, in the studios where they mix cameras, videos may not even be in sync. Even internally, they don’t have this because people think that “realtime” is good enough. That also means that people are not interested in fixing it. Frustrating. We rather interpret “real-time” as “no control on time”. Also example of a live show recorded with a timing signal coming from some MP3, with a photographer recording from within the public. Ingar: We noticed that there have been lots of talks where synchronization is mentioned. It’s a recurring theme, right? Chris: That’s correct. We have a long thread on synchronization of captions for instance. Precise sync is typically needed to handle scene changes. One of the browser engineers from Chrome commented that the JavaScript execution and the video playback are decoupled anyway, so there will always be some inaccuracy. Even if the browser knows things precisely, by the time it surfaces that to the application, some inaccuracy may have occurred. That’s why I was thinking that, potentially, the WebCodecs approach solves some of that, since you move things to the application. Recurring discussion but it had not really gone anywhere. One thing that I am reminded of is the Professional Media Production community who are trying to develop video editors on the web. Synchronization is more crucial there. Ingar: In the object based media initiative at BBC, there are use cases where synchronization is important too. We’ve seen frameworks for video as well. Syncing stuff with the video element. Illustrative of the same idea. Chris: Yes, we were involved in an EU research project called 2IMMERSE. They somewhat re-invented TimingObject from a different angle, also better integration with HbbTV. I can well imagine that they bumped into the same issues that you raised. I don’t know to what level of details they went though. Ingar: The Apple proposal is less about timing and more about social shared viewing. Chris: Yes, handling commands under the hoods. Njaal: Most of the problems arise because things are reported after the fact. If you’re using an external clock, you know when things will happen, so you can prepare. If you know what is going to happen, you can prepare things in advance. Ingar: Also useful to transfer clocks between processes where you can account for latency. Njaal: Most of this stuff could be solved if performance.now was reported along with currentTime. Christoph: That is something that the Web Audio API does. Njaal: Similar to requestVideoFrameCallback, with some useful timing information. If the timeupdate event had the same kind of info, it would be useful. [Edit: it now seems to have] I think all events on the Web should have a “performance.now” measure. Ingar: Christoph, I think you’ve done a parallel implementation of TimingObject and TimingProvider. Also, you have been advocating the approach at Demux and other occasions. Christoph: Yes, the reason I’m excited about the spec, is that people keep redoing this, a bit differently. Mostly done by people that have this one use case in mind. I think it would be cool to have a generic solution. I remember that there was a Web Audio API discussion around implementation support across 2 browsers to push the spec forward. Francois: We’re not on the Recommendation track, so it doesn’t matter right now. We would need two separate implementations (although I note WebRTC uses the same library under the hood). When it’s an API targeted at browsers, we’d need to make sure they’re fine with the API even if they don’t implement it. So would need at least one of the major browsers to be in support. For TimingObject, if we think it doesn’t need browsers and is more a library, we can progress if there are multiple library implementations. Useful as evidence if we want to move to a Working Group to standardise. I don’t think we’re there yet, we’re a small community without a lot of time. Are you willing to lead on writing and updating the spec? Christoph: I’m happy to do it, but I’m new to spec editing. Francois: We can help with that, introduce the tooling, etc. But would it be time well spent? Christoph: When I present on this, I get feedback to say that people are interested. But not necessarily enough to contribute, they’d be happy just to use it. Francois: Second Screen WG was working on a spec called Multi Screen Window Placement, renamed it to Window Management, they dropped the Multi Screen part. Should we rename to Web Timing, as it’s also be about single device use cases? Ingar: Or, media timing. Christop: Feedback I got is some people didn’t know what it was for. But explaining it’s for multi device timing made them realise, but also set a limit on how they saw it. Ingar: Are there other timing efforts in W3C? Chris: There’s the Performance.now spec in Web Performance WG Christoph: If you Google for Web Timing you get info about page loads Francois: That’s the Web Performance group Ingar: I’m fine with changing the name, also if Christoph would like to edit the spec. The spec itself could use updating anyway Christoph: I’ll do what I can but I can’t promise Ingar: This is presented as an interoperability thing. Once you get into this, you quickly realize that there are many approaches to things, e.g. extensions Christoph suggested. That is something that I also see as a problem. Draft should remain simple and not commit to too many of these extra features (until benefit and need is demonstrated). There is a conflict as people tend to think in terms of applications. Shaping things to one application immediately is useful but constrained. Francois: That brings a discussion about the exact features to include in the spec. Christoph: If the consumer wants a smooth clock isn’t something the timing provider knows. Whether those things should be in the spec is debatable. Ingar: A part that’s least mature, there’s also the idea to have access restriction on objects, so some have read only access. That could be a property of the object, but it’s not in the spec. These are all open questions. The impact on the TimingProvider and TimingObject could be to make the TimingObject more complicated. Francois: Wrapping up, I’m hearing interest from Christoph to try to help. That’s good. I don’t have a magic solution to get interest from the browser vendors. We discussed some ideas worth exploring, maybe to capture in issues: an ACID-like test, developing test suites, prototyping using WebCodecs. Ingar: Vimeo have signaled that they’re in the process of adding support for their player. They could be finding the same issues with media element sync. Francois: The media players could be the most important implementers, Vimeo, Shaka Player, etc. If the technology is used in actual products with larger user bases, it gives incentives to discuss more concrete next steps, as there’s use across a wider set of applications. Ingar: Are you in contact with Vimeo, Christoph? Christoph: Yes, someone from the Vimeo team told me about it. Ingar: We’ve been using this stuff for 10 years, not just in media, but also rescue scenarios, there are lots of use cases. The sequencer library we made doesn’t have the same dependencies, so works well and we use it all the time. Ingar: We won’t schedule a next meeting, but can do if there’s something new to discuss.
Received on Thursday, 6 April 2023 13:53:07 UTC