- From: Francois Daoust <fd@w3.org>
- Date: Mon, 26 Sep 2016 10:53:55 +0200
- To: <public-webtiming@w3.org>
Hi Multi-Device Timing CG,
The minutes of last week's CG meeting at TPAC are available at:
https://www.w3.org/2016/09/22-sync-minutes.html
... and copied as raw text below.
The group discussed timing challenges, media sync support in current browsers, proposed solutions and improvements to the Web platform as well as possible next steps.
Thanks,
Francois.
-----
Multi-Device Timing CG F2F
22 Sep 2016
[2]Agenda
[2] https://lists.w3.org/Archives/Public/public-webtiming/2016Sep/0003.html
See also: [3]IRC log
[3] http://www.w3.org/2016/09/22-sync-irc
Attendees
Present
Ingar_Arntzen(MotionCorp), Francois_Daoust(W3C),
Mike_Assenti(Dolby), Louay_Bassbouss(Fraunhofer),
Chris_Needham(BBC), Kumanan_Yogaratnam(Espial),
Toshihiko_Yamakami(Access), Anssi_Kostiainen(Intel),
Louay_Bassbouss, Shih-Chiang_Chien(Mozilla),
Eric_Carlson(Apple)
Chair
Ingar
Scribe
Chris_Needham
Contents
* [4]Topics
1. [5]Introduction
2. [6]Timing Challenges
3. [7]Goals for the group
4. [8]Proposal
5. [9]Standardisation
6. [10]Related work
7. [11]Next steps
__________________________________________________________
Ingar: This is the first F2F of the Multi-Device Timing CG
... This work comes from the MediaScape EU project
... The focus is on multi-device timing across web pages, and
providing a programming model to deal with temporal aspects in
web apps
... This is a multi-device presentation
-> [12]Slideshow
[12] https://goo.gl/AlR9Nm
Ingar: I'd like to show a [13]demo
... [ demonstrates showing a video on several devices in sync ]
... If you look at the edges between the browsers, this is
where you'll see any sync errors
... The synchronisation was between two browser windows on a
single machine
... You can achieve similar precision in the global scope, eg,
between devices in this room
[13] http://mcorp.no/pres/vegas16/main.html
Toshihiko: What would happen when I return home to Japan?
Ingar: ... It would still play in sync
... It's symmetric, in the sense that we all have control
... [ demonstration of pausing the video, all devices pause ]
... I think we're in the single-digit millisecond region
... The play and pause control commands do have a day, may take
0.2 seconds or so
Mike: Are these commands also part of the spec?
Ingar: Yes
... We also handle re-establishing sync on page reload
Introduction
Ingar: Francois presented at TPAC last year
... the draft Timing Object spec
... using an online timing provider from Motion Corp
... Since then we have an open source JavaScript implementation
... We've demonstrated at NAB and IBC, and also published some
papers
... Media Element capabilities have also progressed
... We were having trouble with Chrome on Android before, but
this has become very good now
... Safari has a bug
Anssi: Has that been reported?
Ingar: We'll be doing that, yes
... Timestamping on the client side, using timing info in
getUserMedia
... We've also reported bugs in Chromium, now fixed
... Multi-device timing is about making stuff happen at the
right time
... It's not only about having a clock, but also interactive
capabilities to control it
... Timing scopes: page internal, within the browser context,
and also cross-page
... Even though the group's name is Multi-Device Timing, both
of these scopes are important
... The Timing Object is a client-side API
... We don't prescribe a particular solution for the cross-page
timing
... It can be implemented in different ways
Timing Challenges
Ingar: With the page internal timing, you could want to
synchronise multiple videos
... Ad insertion can be done with MSE
Francois: There are some limitations with what you can do with
MSE
Ingar: Our approach gives more flexibility
... It works with different types and frameworks
... iframes are important, eg for independently delivered
adverts
... You can split content across windows or frames
... You can also sync between web pages and native apps
... Collaborative applications
... It can work across different types of network connection,
eg, 3G or wifi
... Media elements and animations have different APIs, making
reusability and flexibility difficult
... The Media element currentTime attribute can be quite coarse
Francois: The spec doesn't force the UA to apply a precise
clock
Anssi: We specified an extension to Web Audio, the context
time, which is when the audio hits the speakers
... We'll be adding timing to sensors, etc, anywhere it makes
sense
... We're working on low-level primitives that make your use
cases possible
... An example is screen touch events, knowing when these
happen
Ingar: It would be nice to write a sync wrapper for Web Audio
... Non-determinism is a problem, especially in the distributed
case
... There's no support for cross-page timing at the moment
... It's not a sound assumption to rely on NTP synchronised
system clocks
... Control and timing are typically dealt with separately, but
we're trying to deal with these in a single abstraction
Francois: Have you thought about how Web Audio synchronisation
might apply to the Presentation API?
Anssi: No, not yet.
... [ some discussion of issues with Performance.now ]
Ingar: We get around that, using a Timing Object, which you can
give to two processes in the same architecture
Goals for the group
Ingar: We can simplify things for developers
... Interoperability is important. With the web everything is
composeable, but not timing
... Also synchronising web and broadcast content
... We have a concrete proposal, and our demos show that it's
feasible
... [ [14]another demo ]
[14] https://youtu.be/D6cBn65KXTk
Ingar: It shows how to create a web page that uses timing
Kumanan: There's an issue here with iOS, where it only allows
one video playing at once
... I have Blink in one and Firefox in the other
Proposal
Ingar: The timing object can play, pause, jump. But it's not
connected to a media element
... [ demo of local timing object ]
... You can change position, velocity, acceleration
... through the API using a vector
Francois: There's the same kind of object being defined in
WebVR, where a pause operation affects velocity
Chris: What's the motiviation for acceleration? Could you have
easing functions for changing between one velocity and another
Ingar: Acceleration isn't a central thing in this. You could
build such functions on top of the API
... It's not critical, but could be useful
Kumanan: Does velocity have direction?
Ingar: Yes, so negative reverses the playback direction
Mike: Are the units arbitrary?
Ingar: Yes.
Mike: If we want to relate this to a media content pipeline,
For example, if you want to double the rate, how do you know
what the scale of the timeline is?
Francois: When you associate the Timing Object with a media
element, position would be seconds, so the units are set there
... We're also thinking about how to compose Timing Objects,
different clocks at different rates
Ingar: There are different options. You could define a mapping,
so we've defined a TimingConverter
... You'd use this with media objects having different
timelines
... You could also change the data to fit the different
timeline
Kumanan: When I change velocity, how do I know what my normal
is?
Mike: I think the question is how the position and velocity
relate to the timestamp value in the vector
Kumanan: The timestamp comes from Performance.now, which is
local
[ discussion of how timestamp relates to the vector ]
Ingar: An application can remember what the normal is. The
velocity doesn't jitter
... The TimingObject has change and timeupdate events
... When someone changes the TimingObject, all others are
updated
... The TimingObject is in control, and it's the component's
responsibility to respond
Toshihiko: Will the TimingObject cover other use cases, such as
a carousel where you want to loop from the end to the start
Ingar: It's quite simple to add that on top of this
... It's a matter of preference
... A hosted internet service can provide a timing service, so
the local TimingObject becomes a proxy
... There's latency but the state will be correct
... One advantage of a hosted solution is availability
... [ demo of online timing object ]
... Doing this in user space JavaScript, you lose some
precisiion
... We have the TimingObject API for clients, but there's also
an API for timing providers
... This is simpler, as it just has to maintain two variables:
the vector, and the skew
... The main source of error is the step between the
TimingObject and the media element.
Kumanan: If this becomes a proper Web API, would there be a
timing provider mandated?
Francois: The default implementation would use Performance.now
... [discussion of timing providers and skew ]
Ingar: The skew would be that between Performance.now and the
underlying timing source
... There are lots of applications: collaborative viewing,
multi-screen displays, multi-device in the home, showing
interactive content with a main video feed
... Also remote control, music, multi-device capture,
timestamping material according to an app-specific timeline
... Which makes the content re-playable in a multi-device setup
... We've done some experiments on sync precision
... We're measuring the difference between the TimingObject and
the media position
... When there are big differences, it's best to skip position
... When it's closer, adjust the playbackRate
... [ shows graphs of measurements taken on Android, Chrome ]
... We'd like to be able to supply a timestamp to the media
element's play command
... [ graph shows problem with playbackRate on iPad ]
Standardisation
Ingar: Improvements needed with playBackRate, seek
Francois: We may also need to look at Web Animation, you may
not know the latency, ie, when the animation will actually
start
Ingar: Text tracks are limiting, we have a sequencer object
that uses timing object
... We use setTimeout in the sequencer, which gives precision
of a few milliseconds, but delays if the browser is busy
Francois: Due to the event loop, but could use tasks or micro
tasks instead
Kumanan: Wet set limits to the setTimeout interval
Related work
Ingar: There's the Second Screen WG
Louay: HbbTV 2.0 has sync, discovery, communication, app launch
in one package
... Similar for Google Cast
... The protocol work in the Second Screen group will touch on
synchronisation
... Should we have one group at W3C looking at this?
Francois: I don't expect the Second Screen WG to take on this
work
... One thing in-scope for the Second Screen CG is reporting of
currentTime for the Remote Playback API
Kumanan: That just needs to address reporting
Francois: I'm trying to gather interest in the multi-device
timing CG
... There have been improvements in the web platform, which
make sync easier, but no specific work on sync itself
Kumanan: I suggest a separate timing spec is valuable more
generally than just media applications
Francois: That's good feedback. I would have thought that
browser vendors would not fancy supporting a timing object
natively if it sits "on the side", meaning if it does not
connect to e.g. media elements. That is the reason why we
included the connection with media elements in the draft spec.
Kumanan: There are applications such as real time collaboration
where this is valuable
Next steps
Ingar: We want to attract support to the CG
Francois: We could track progress in the existing specs that
have timing involved
Kumanan: What's the blocker to moving this forward?
Francois: It needs someone willing to implement natively
... [ discussion on differences between CGs, WGs, and IGs at
W3C ]
Ingar: [ recaps issues with iOS timing measurements ]
Louay: Do you have any experience of syncing live streams, eg
HLS?
Ingar: Yes, i have a demo
Louay: [ discussion of currentTime in live streams ]
Eric: startDate should work with HLS streams if the information
is in the manifest
... [ discussion of the play method that returns a Promise ]
[End of minutes]
Received on Monday, 26 September 2016 08:54:13 UTC