Re: Review of the Timing Object specification

Thanks Paul!

I find you comments very constructive and I'd like to compare them to the
current spec. Please let me know if I end up misrepresenting any of your
ideas.


   - Timing and synchronization needs to be capable of going both ways
   between interacting objects

I think we got this one covered. The timing object supports symmetric
control through its update method. It can also do asymmetric master-slave
(only master does update)


   - Temporal controls need to at least include absolute time references
   (central clock), relative time references (local clock) and event based
   timing (triggers.)

It might not be entirely clear in the spec, but timing object is as you say
based on
1) local clock (performance.now())
2) central clock (clock used by timing provider) defined by local clock +
vector [skew, drift, timestamp_localclock]
3) timing object (central clock + vector [position, velocity, acceleration,
timestamp_centralclock])

In the spec the central clock is currently hidden - but in the timingsrc
implementation I've chosen to expose it with the timingobject.clock
property.
This is particularly helpful with respect to the timeout mechanism I use
(wrapper around setTimeout) which needs to be relative to the central clock
- not the local clock.

I will propose adding timingobject.clock property to the spec plus and API
for this clock.

With regard to event-based timing triggers - the timingsrc library
currently includes both timedPointCallbacks and Sequencers for this
purpose. I view them not as an integral part of the timing object - but as
tools to use with the timing object.


   - Multi-screen is the tip of the iceberg. Integrated virtual and
   physical realities are more transparent to the participants and should be
   considered, hence real world objects/actors need to have the ability to be
   represented digitally and included in the synchronization of the user
   experience


Fully agree. You are more than welcome to contribute a use case along these
lines to the spec.


   - Fail down from most to least desirable timing method should be
   available

This is something that could be supported by a timing provider. For
instance, knowledge about NTP synchronized clocks and estimated accuracy
would be helpful. Fail down to on-demand clock sync if NTP is not available
or not recently synchronized etc.


   - Just because there is some way to do something today doesn’t mean that
   we shouldn’t include that capability in a simpler, easier form, e.g. there
   are some temporal functions that can be done with JavaScript today, but why
   condemn tomorrow’s programmers to fuss with that if those capabilities
   could be built into HTML5+? One could always resort to JavaScript if they
   need something different, but catch the most common 80% if you can.

I agree with this, and see it as relevant to discussions like
- should the timing object look more like the media api?
- should we prioritize backwards compatibility with the <track> mechanism
or argue standardization and built-in support for precise sequencing?


   - Temporal controls should not just be for temporal media (e.g. video
   and audio) but for any rich media object, and also for code execution and
   device control. The concept of multi-[whatever] should not be limited to
   screens but include devices, windows on the same device, applications on
   the same or other device(s,) locations, incidences of the same events on
   different devices with different users, etc.


Absolutely! This perspective has motivated a clean, yet very expressive
design for the timing object. For instance, high speeds, reverse or
acceleration might not be convenient nor relevant in the context of AV
media - but it might make sense in other products and other domains. It's
important to realize that applications supporting the timing object API do
NOT have to support all possible states of the timing object. For instance,
AV media players might default to doing nothing while the timing object is
in an unsupported state, and then resuming correct behavior when the timing
object goes back to a supported state.


   - It would be good for a timing offset to be specifiable, both manually
   in code and via an API into a given timing object. In conjunction with
   this, there could be an auto-synch function (in the timing object or
   separately called) which would test the network delay between participants
   in the user experience and correct for delay/latency. This could be
   triggered by code or set to periodically retest.

Not sure I understand the difference between "manually in code" and "via
and API". Do you mean specifying timing offset declaratively in a tag?

Collecting the network delay of multiple agents in a distributed system may
be nice, as you would know the latency of the "slowest" agent, and you
could then use this to schedule distributed timed events cleverly. However,
it is not something that is required by all applications, and it is not
required for correct distributed synchronization of timing objects. It is
also potentially a limiting factor on scalability. For these reasons we
have opted to not include this in the timing object spec. That said, the
timing object could expose its latency measurements, thus allowing
applications programmers to choose an independent method for communicating
this information among agents.


   - The timing object API should allow for a script to be specified by
   URI, which the timing object would use to push complex timings to the
   objects in the user experience. The script could include the
   parameters/arguments that the Timing Object accepts via API and place those
   along a timing reference. A script could also allow for calls to
   executables and importing of other scripts, at points along the time
   reference used.

If I understand you correctly, this is about timed scripting/control of the
timing object itself.

I've got a few use cases for this. For instance, if you want to change
speed like an animation - accelerate for a short while, then decelerate and
at some point turn off acceleration. We have discussed the introduction of
a more sophisticated update function for this purpose - effectively
allowing a sequence of timed update requests to be scheduled for delayed
execution. This is absolutely doable, though I don't see it is critical for
v1, so this could be an extension at a later point (v2?).

Another use case would be to timestamp interaction with the timing object,
with the intent of replaying this interaction correctly at later time. We
have done this on a couple of occasions, and I don't think this requires
any additional support from the timing object as this is exactly what the
Sequencer is for. The Sequencer can be used for any kind of timed activity,
including timed calls to executables or timed imports of new scripts. See
webtiming.github.io/sequencer/ for more info (will be migrated to
webtiming.github.io/timingsrc/ shortly)


Again, thank you for this great feedback Paul.

Best regards,

Ingar




2015-12-01 15:50 GMT+01:00 GAUSMAN, PAUL <pg2483@att.com>:

> Personally, I don’t think it matters if there are any media players that
> can immediately use these temporal functions because we are talking about
> the future here, and any code publisher who wants to use temporal functions
> will enhance their product to take advantage of good temporal functions
> when they are available.
> I would offer these comments:
>
>    - Timing and synchronization needs to be capable of going both ways
>    between interacting objects
>    - Temporal controls need to at least include absolute time references
>    (central clock), relative time references (local clock) and event based
>    timing (triggers.)
>    - Multi-screen is the tip of the iceberg. Integrated virtual and
>    physical realities are more transparent to the participants and should be
>    considered, hence real world objects/actors need to have the ability to be
>    represented digitally and included in the synchronization of the user
>    experience
>    - Fail down from most to least desirable timing method should be
>    available
>    - Just because there is some way to do something today doesn’t mean
>    that we shouldn’t include that capability in a simpler, easier form, e.g.
>    there are some temporal functions that can be done with JavaScript today,
>    but why condemn tomorrow’s programmers to fuss with that if those
>    capabilities could be built into HTML5+? One could always resort to
>    JavaScript if they need something different, but catch the most common 80%
>    if you can.
>    - Temporal controls should not just be for temporal media (e.g. video
>    and audio) but for any rich media object, and also for code execution and
>    device control. The concept of multi-[whatever] should not be limited to
>    screens but include devices, windows on the same device, applications on
>    the same or other device(s,) locations, incidences of the same events on
>    different devices with different users, etc.
>    - It would be good for a timing offset to be specifiable, both
>    manually in code and via an API into a given timing object. In conjunction
>    with this, there could be an auto-synch function (in the timing object or
>    separately called) which would test the network delay between participants
>    in the user experience and correct for delay/latency. This could be
>    triggered by code or set to periodically retest.
>    - The timing object API should allow for a script to be specified by
>    URI, which the timing object would use to push complex timings to the
>    objects in the user experience. The script could include the
>    parameters/arguments that the Timing Object accepts via API and place those
>    along a timing reference. A script could also allow for calls to
>    executables and importing of other scripts, at points along the time
>    reference used.
>
>
> Thanks!
>
> Paul
>
> *From:* Ingar Mæhlum Arntzen [mailto:ingar.arntzen@gmail.com
> <ingar.arntzen@gmail.com>]
> *Sent:* Tuesday, December 01, 2015 6:52 AM
> *To:* Igarashi, Tatsuya
> *Cc:* fd@w3.org; public-web-and-tv IG; Ingar Mæhlum Arntzen; Njaal Borch
> *Subject:* Re: Review of the Timing Object specification
>
> Dear Tatsuya Igarashi
>
> Thank you !
>
> These two examples represent different approaches to similar goals.
>
> In example X you ask the video to play - an it plays in its own timframe
> (progression defined ultimately by the media stack).
> In example 6 you specify a timing object as timeframe (progression
> ultimately defined by performance.now() or a timing provider) and asks the
> video to align itself.
>
> If all you want to do is to play a video in a single video element - then
> this distinction is of marginal interest.
> However, if you want to time-align video playback with something else - in
> the same web page or on another device, this distinction becomes important.
> You may read more about this distinction in the timing object spec under
> "timed playback mode" [1]
>
> We have demonstrated that it is somewhat possible in JavaScript to force a
> video element to be time-aligned with a timing object. This works best when
> variable playbackrate is supported by the video element. The solution is
> based on continuously monitoring the diff between timing object and
> currentTime of video element, and performing appropriate adjustments. You
> can check out the code here [2]
>
> We also have published a report [3] on how well media elements currently
> responds to such "abuse". However, in order to improve further, browser
> support is required.
>
> You may also verify example 6 in your own browser by opening this demo [4]
> of two videos synchronized by the timing object. JS libraries are available
> in the timingsrc repository [5]
>
> [1]
> *http://webtiming.github.io/timingobject/#media-elements-and-the-timing-object*
> <http://webtiming.github.io/timingobject/>
> [2]
> *https://github.com/webtiming/timingsrc/blob/master/source/mediasync/mediasync.js*
> <https://github.com/webtiming/timingsrc/blob/master/source/mediasync/mediasync.js>
> [3]
> *https://lists.w3.org/Archives/Public/public-webtiming/2015Oct/0003.html*
> <https://lists.w3.org/Archives/Public/public-webtiming/2015Oct/0003.html>
> [4]
> *https://github.com/webtiming/timingsrc/blob/master/examples/mediasync.html*
> <https://github.com/webtiming/timingsrc/blob/master/examples/mediasync.html>
> [5] *https://github.com/webtiming/timingsrc*
> <https://github.com/webtiming/timingsrc>
>
>
> Hope this is clarifying :)
>
> Best regards,
>
> Ingar Arntzen
>
>
>
>
>
>
>
> 2015-12-01 11:26 GMT+01:00 Igarashi, Tatsuya <
> *Tatsuya.Igarashi@jp.sony.com* <Tatsuya.Igarashi@jp.sony.com>>:
> Hi,
>
> Media Timeline Synchronize is one of the most interesting topics in Media
> industry.
>
> I wonder that the following two examples are the same semantically. Most
> media player of browsers do not support any of playback speed rates.  Also,
> execution of javascript code is not in real-time. How does the Time Object
> proposal improve accuracy of timeline synchronization of HTML5 media?
>
> Example 6: Use timing object to control a video
> var video; // Pointer to a video element in the DOM
> var to = new TimingObject();
> video.timingsrc = to; // Video now directed by timing object
>
> // Start timing object (video seeks and starts by implication)
> to.update({ position: 5.0, velocity: 1.0 });
>
> Example X : synchronize media timeline of a video
> var video; // Pointer to a video element in the DOM
>
> // Sync media timeline (video seeks and starts with the rate)
> video.currentTime=5.0;video.playbackRate=1.0;video.play();
>
> -***---***---***---***---***---***---***---***---***--***---***---***-
> Tatsuya Igarashi (*Tatsuya.Igarashi@jp.sony.com*
> <Tatsuya.Igarashi@jp.sony.com>)
> Innovative Technology Development Div, System R&D Group
> Sony Corporation
>
>
> -----Original Message-----
> From: Francois Daoust [mailto:*fd@w3.org* <fd@w3.org>]
> Sent: Tuesday, December 01, 2015 1:22 AM
> To: 'public-web-and-tv IG'
> Cc: 'Ingar Mæhlum Arntzen'; 'Njaal Borch'
> Subject: Review of the Timing Object specification
>
> Hi Web and TV IG participants,
>
> Back at TPAC in Sapporo, I presented the Timing Object specification,
> developed by the Multi-Device Timing Community Group to enable single- and
> cross-device synchronization scenarios. I took an action to send the draft
> spec to the Web and TV IG for review.
>
> The Multi-Device Timing Community Group would welcome your feedback on the
> spec by 8 January 2016 (so that it can decide on next steps early next
> year). The latest draft is available for review at:
>
>   http://webtiming.github.io/timingobject/
>
> The Multi-Device Timing Community Group does not anticipate any major
> change to the specification by January. Main open issues are referenced
> from within the spec itself to ease review from an external perspective.
> The group uses GitHub to track issues otherwise:
>
>  https://github.com/webtiming/timingobject/issues
>
> At this stage, the CG is particularly interested in:
>
> 1. discovering Web and TV use cases and requirements that the spec may not
> properly address 2. discussing the proposed mechanism to associate a timing
> object to an external timing resource through code provided by a
> third-party timing resource provider; 3. evaluating the main technical
> hurdles from an implementation perspective.
>
> The CG also invites all interested parties to join the Multi-Device Timing
> CG and let themselves known.
>
> Thanks,
> Francois.
>
> PS: Some of you wondered at TPAC whether there existed an open-source
> prototype implementation of the spec in JavaScript. Ingar and Njaal have
> been working on it, see announcement today at:
>
> https://www.w3.org/community/webtiming/2015/11/30/first-implementation-of-timing-object-draft-spec/
>
>
>
>
>
>