W3C home > Mailing lists > Public > public-webtiming@w3.org > September 2016

[minutes] Multi-Device Timing CG F2F - TPAC Lisbon - 2016-09-22

From: Francois Daoust <fd@w3.org>
Date: Mon, 26 Sep 2016 10:53:55 +0200
To: <public-webtiming@w3.org>
Message-ID: <009f01d217d3$84430a10$8cc91e30$@w3.org>
Hi Multi-Device Timing CG,

The minutes of last week's CG meeting at TPAC are available at:

 https://www.w3.org/2016/09/22-sync-minutes.html

... and copied as raw text below.

The group discussed timing challenges, media sync support in current browsers, proposed solutions and improvements to the Web platform as well as possible next steps.

Thanks,
Francois.


-----
Multi-Device Timing CG F2F

22 Sep 2016

   [2]Agenda

      [2] https://lists.w3.org/Archives/Public/public-webtiming/2016Sep/0003.html

   See also: [3]IRC log

      [3] http://www.w3.org/2016/09/22-sync-irc

Attendees

   Present
          Ingar_Arntzen(MotionCorp), Francois_Daoust(W3C),
          Mike_Assenti(Dolby), Louay_Bassbouss(Fraunhofer),
          Chris_Needham(BBC), Kumanan_Yogaratnam(Espial),
          Toshihiko_Yamakami(Access), Anssi_Kostiainen(Intel),
          Louay_Bassbouss, Shih-Chiang_Chien(Mozilla),
          Eric_Carlson(Apple)

   Chair
          Ingar

   Scribe
          Chris_Needham

Contents

     * [4]Topics
         1. [5]Introduction
         2. [6]Timing Challenges
         3. [7]Goals for the group
         4. [8]Proposal
         5. [9]Standardisation
         6. [10]Related work
         7. [11]Next steps
     __________________________________________________________

   Ingar: This is the first F2F of the Multi-Device Timing CG
   ... This work comes from the MediaScape EU project
   ... The focus is on multi-device timing across web pages, and
   providing a programming model to deal with temporal aspects in
   web apps
   ... This is a multi-device presentation

   -> [12]Slideshow

     [12] https://goo.gl/AlR9Nm

   Ingar: I'd like to show a [13]demo
   ... [ demonstrates showing a video on several devices in sync ]
   ... If you look at the edges between the browsers, this is
   where you'll see any sync errors
   ... The synchronisation was between two browser windows on a
   single machine
   ... You can achieve similar precision in the global scope, eg,
   between devices in this room

     [13] http://mcorp.no/pres/vegas16/main.html

   Toshihiko: What would happen when I return home to Japan?

   Ingar: ... It would still play in sync
   ... It's symmetric, in the sense that we all have control
   ... [ demonstration of pausing the video, all devices pause ]
   ... I think we're in the single-digit millisecond region
   ... The play and pause control commands do have a day, may take
   0.2 seconds or so

   Mike: Are these commands also part of the spec?

   Ingar: Yes
   ... We also handle re-establishing sync on page reload

Introduction

   Ingar: Francois presented at TPAC last year
   ... the draft Timing Object spec
   ... using an online timing provider from Motion Corp
   ... Since then we have an open source JavaScript implementation
   ... We've demonstrated at NAB and IBC, and also published some
   papers
   ... Media Element capabilities have also progressed
   ... We were having trouble with Chrome on Android before, but
   this has become very good now
   ... Safari has a bug

   Anssi: Has that been reported?

   Ingar: We'll be doing that, yes
   ... Timestamping on the client side, using timing info in
   getUserMedia
   ... We've also reported bugs in Chromium, now fixed
   ... Multi-device timing is about making stuff happen at the
   right time
   ... It's not only about having a clock, but also interactive
   capabilities to control it
   ... Timing scopes: page internal, within the browser context,
   and also cross-page
   ... Even though the group's name is Multi-Device Timing, both
   of these scopes are important
   ... The Timing Object is a client-side API
   ... We don't prescribe a particular solution for the cross-page
   timing
   ... It can be implemented in different ways

Timing Challenges

   Ingar: With the page internal timing, you could want to
   synchronise multiple videos
   ... Ad insertion can be done with MSE

   Francois: There are some limitations with what you can do with
   MSE

   Ingar: Our approach gives more flexibility
   ... It works with different types and frameworks
   ... iframes are important, eg for independently delivered
   adverts
   ... You can split content across windows or frames
   ... You can also sync between web pages and native apps
   ... Collaborative applications
   ... It can work across different types of network connection,
   eg, 3G or wifi
   ... Media elements and animations have different APIs, making
   reusability and flexibility difficult
   ... The Media element currentTime attribute can be quite coarse

   Francois: The spec doesn't force the UA to apply a precise
   clock

   Anssi: We specified an extension to Web Audio, the context
   time, which is when the audio hits the speakers
   ... We'll be adding timing to sensors, etc, anywhere it makes
   sense
   ... We're working on low-level primitives that make your use
   cases possible
   ... An example is screen touch events, knowing when these
   happen

   Ingar: It would be nice to write a sync wrapper for Web Audio
   ... Non-determinism is a problem, especially in the distributed
   case
   ... There's no support for cross-page timing at the moment
   ... It's not a sound assumption to rely on NTP synchronised
   system clocks
   ... Control and timing are typically dealt with separately, but
   we're trying to deal with these in a single abstraction

   Francois: Have you thought about how Web Audio synchronisation
   might apply to the Presentation API?

   Anssi: No, not yet.
   ... [ some discussion of issues with Performance.now ]

   Ingar: We get around that, using a Timing Object, which you can
   give to two processes in the same architecture

Goals for the group

   Ingar: We can simplify things for developers
   ... Interoperability is important. With the web everything is
   composeable, but not timing
   ... Also synchronising web and broadcast content
   ... We have a concrete proposal, and our demos show that it's
   feasible
   ... [ [14]another demo ]

     [14] https://youtu.be/D6cBn65KXTk

   Ingar: It shows how to create a web page that uses timing

   Kumanan: There's an issue here with iOS, where it only allows
   one video playing at once
   ... I have Blink in one and Firefox in the other

Proposal

   Ingar: The timing object can play, pause, jump. But it's not
   connected to a media element
   ... [ demo of local timing object ]
   ... You can change position, velocity, acceleration
   ... through the API using a vector

   Francois: There's the same kind of object being defined in
   WebVR, where a pause operation affects velocity

   Chris: What's the motiviation for acceleration? Could you have
   easing functions for changing between one velocity and another

   Ingar: Acceleration isn't a central thing in this. You could
   build such functions on top of the API
   ... It's not critical, but could be useful

   Kumanan: Does velocity have direction?

   Ingar: Yes, so negative reverses the playback direction

   Mike: Are the units arbitrary?

   Ingar: Yes.

   Mike: If we want to relate this to a media content pipeline,
   For example, if you want to double the rate, how do you know
   what the scale of the timeline is?

   Francois: When you associate the Timing Object with a media
   element, position would be seconds, so the units are set there
   ... We're also thinking about how to compose Timing Objects,
   different clocks at different rates

   Ingar: There are different options. You could define a mapping,
   so we've defined a TimingConverter
   ... You'd use this with media objects having different
   timelines
   ... You could also change the data to fit the different
   timeline

   Kumanan: When I change velocity, how do I know what my normal
   is?

   Mike: I think the question is how the position and velocity
   relate to the timestamp value in the vector

   Kumanan: The timestamp comes from Performance.now, which is
   local

   [ discussion of how timestamp relates to the vector ]

   Ingar: An application can remember what the normal is. The
   velocity doesn't jitter
   ... The TimingObject has change and timeupdate events
   ... When someone changes the TimingObject, all others are
   updated
   ... The TimingObject is in control, and it's the component's
   responsibility to respond

   Toshihiko: Will the TimingObject cover other use cases, such as
   a carousel where you want to loop from the end to the start

   Ingar: It's quite simple to add that on top of this
   ... It's a matter of preference
   ... A hosted internet service can provide a timing service, so
   the local TimingObject becomes a proxy
   ... There's latency but the state will be correct
   ... One advantage of a hosted solution is availability
   ... [ demo of online timing object ]
   ... Doing this in user space JavaScript, you lose some
   precisiion
   ... We have the TimingObject API for clients, but there's also
   an API for timing providers
   ... This is simpler, as it just has to maintain two variables:
   the vector, and the skew
   ... The main source of error is the step between the
   TimingObject and the media element.

   Kumanan: If this becomes a proper Web API, would there be a
   timing provider mandated?

   Francois: The default implementation would use Performance.now
   ... [discussion of timing providers and skew ]

   Ingar: The skew would be that between Performance.now and the
   underlying timing source
   ... There are lots of applications: collaborative viewing,
   multi-screen displays, multi-device in the home, showing
   interactive content with a main video feed
   ... Also remote control, music, multi-device capture,
   timestamping material according to an app-specific timeline
   ... Which makes the content re-playable in a multi-device setup
   ... We've done some experiments on sync precision
   ... We're measuring the difference between the TimingObject and
   the media position
   ... When there are big differences, it's best to skip position
   ... When it's closer, adjust the playbackRate
   ... [ shows graphs of measurements taken on Android, Chrome ]
   ... We'd like to be able to supply a timestamp to the media
   element's play command
   ... [ graph shows problem with playbackRate on iPad ]

Standardisation

   Ingar: Improvements needed with playBackRate, seek

   Francois: We may also need to look at Web Animation, you may
   not know the latency, ie, when the animation will actually
   start

   Ingar: Text tracks are limiting, we have a sequencer object
   that uses timing object
   ... We use setTimeout in the sequencer, which gives precision
   of a few milliseconds, but delays if the browser is busy

   Francois: Due to the event loop, but could use tasks or micro
   tasks instead

   Kumanan: Wet set limits to the setTimeout interval

Related work

   Ingar: There's the Second Screen WG

   Louay: HbbTV 2.0 has sync, discovery, communication, app launch
   in one package
   ... Similar for Google Cast
   ... The protocol work in the Second Screen group will touch on
   synchronisation
   ... Should we have one group at W3C looking at this?

   Francois: I don't expect the Second Screen WG to take on this
   work
   ... One thing in-scope for the Second Screen CG is reporting of
   currentTime for the Remote Playback API

   Kumanan: That just needs to address reporting

   Francois: I'm trying to gather interest in the multi-device
   timing CG
   ... There have been improvements in the web platform, which
   make sync easier, but no specific work on sync itself

   Kumanan: I suggest a separate timing spec is valuable more
   generally than just media applications

   Francois: That's good feedback. I would have thought that
   browser vendors would not fancy supporting a timing object
   natively if it sits "on the side", meaning if it does not
   connect to e.g. media elements. That is the reason why we
   included the connection with media elements in the draft spec.

   Kumanan: There are applications such as real time collaboration
   where this is valuable

Next steps

   Ingar: We want to attract support to the CG

   Francois: We could track progress in the existing specs that
   have timing involved

   Kumanan: What's the blocker to moving this forward?

   Francois: It needs someone willing to implement natively
   ... [ discussion on differences between CGs, WGs, and IGs at
   W3C ]

   Ingar: [ recaps issues with iOS timing measurements ]

   Louay: Do you have any experience of syncing live streams, eg
   HLS?

   Ingar: Yes, i have a demo

   Louay: [ discussion of currentTime in live streams ]

   Eric: startDate should work with HLS streams if the information
   is in the manifest
   ... [ discussion of the play method that returns a Promise ]

   [End of minutes]
Received on Monday, 26 September 2016 08:54:13 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:25:15 UTC