Re: Minutes from Media Timed Events Task Force call, 20 August 2018

Hi Chris, Francois and all.

In reference to the discussion Media Timed Events Task Force call, 20
August 2018 [1].

- it seems the sequencer [2] provides a relevant model for solution here --
again ;)

>>>
Francois: I'm trying to understand how the event model differs from the
usual Web eventing model.
.... I understand your mode 2 is similar to the media element's event
model, but mode 1 differs.
Charles: Yes, emsg events could arrive at the beginning of a segment. The
emsg box describes the start time and duration
.... Notifying the application when the event is received gives it
additional time.
Will: I can confirm that. We have a preference for mode 1, so that we can
put the events into a helper library to dispatch at the right time.
>>>

- We have worked a lot with timed, interactive data, though not
specifically in the context of dash.
  We always separate
    1) data availability (data for given portion of the timeline -- live
data as soon as possible) and
    2) data presentation (at the right time).
We typically solve 1) by maintaining (client-side) a collection of received
timed data, and then the sequencer [2] monitors that collection to solve 2)
-- essentially providing another collection (subset) representing the
active data. This gives us both mode 1 and 2, and we often need them both.

>>>
Francois: There's no event when a change is made to the list of cues, even
when not active.
.... Maybe it's just a matter of adding an event when the list changes.
>>>

- We detect all content changes by monitoring the collection of available
data from 1).  In addition, the sequencer gives events for all content
changes that relate to active data 2) .

>>>
Chris: You say the event carries a start time and a duration. Is it
possible for an event to be unbounded?
.... So you don't know when it ends in the future.
Giri: Internally, we've struggled with TTML on ways to define an unbounded
metadata cue. "Display once, keep it on the display until I tell you to get
rid of it". We haven't found something in TextTrack that lets us do that
out of the box.
>>>

- the sequencer supports unbounded time intervals, as well as dynamically
changing time intervals. I guess the open question is whether the dash
transport mechanism would handle the update of timing information for a
given item, and if so what the delay would be?

>>>
Chris: [...]
.... Another question: If we were to push for a Web platform API to do
this, is the preference still for the Mode 1? As I understand it, the
current model that we have, using TextTracks, is closer to Mode 2.
Chris: Interesting that we don't have something for mode 1 on the Web.
>>>

- that's perhaps because sadly there has not been a strong tradition in the
Web for separating media transport from media control and media
presentation? The media player just does it all.

>>>
Will: [...] We have a preference for mode 1, so that we can put the events
into a helper library to dispatch at the right time.
>>>

- we share this preference I guess. From the perspective of precise timing
and synchronization, it might even be beneficial [3] to avoid doing the
timing of interactive data within the media player, and instead do it in
the application as Will suggests (assuming media clock of media player is
available in application).


Best, Ingar Arntzen

[1] http://lists.w3.org/Archives/Public/public-web-and-tv/2018Aug/0018.html
[2] https://webtiming.github.io/timingsrc/doc/api_sequencer.html
[3]
https://github.com/w3c/media-and-entertainment/issues/4#issuecomment-398163071



Den ons. 22. aug. 2018 kl. 11:24 skrev Chris Needham <
chris.needham@bbc.co.uk>:

> Dear all,
>
> The minutes from yesterday's Media Timed Events Task Force call are
> available [1], and copied below. Many thanks to Charles for presenting.
>
> The next Task Force call will be on Monday 17th September.
>
> Kind regards,
>
> Chris (Co-chair, W3C Media & Entertainment Interest Group)
>
> [1] https://www.w3.org/2018/08/20-me-minutes.html
>
> ---
>
> W3C
> - DRAFT -
> Media and Entertainment IG - Media Timed Events TF
>
> 20 Aug 2018
>
> Agenda
>
> https://lists.w3.org/Archives/Public/public-web-and-tv/2018Aug/0015..html
>
> Attendees
>
> Present
>         Kaz_Ashimura, Giri_Mandyam, Charles_Lo, Chris_Needham,
> Ali_C_Begen, John_Luther, Steve_Morris, Will_Law, Francois_Daoust
>
> Regrets
>
> Chair
>         Giri
>
> Scribe
>         cpn, tidoust
>
> Contents
>
> Topics
>
>         Introduction
>         Background
>         Architecture
>         APIs
>         DASH processing model
>         Work status
>         Questions and discussion
>         Timing and synchronization
>
> Summary of Action Items
> Summary of Resolutions
>
> <cpn> scribenick: cpn
>
> Introduction
>
> Giri: I asked my colleague Charles Lo to present an initiative from
> DASH-IF..
> .... It's work in progress, so we can only go into so much detail.
> .... We'll distribute the slides after the call.
>
> Charles: Thanks, Giri
> .... I'm Charles, from Qualcomm.
>
> <kaz> Charles's slides on DAInty API:
> https://www.w3.org/2011/webtv/wiki/images/4/44/DASH-IF_DAInty_API_Overview_August_20_2018.pdf
>
> Charles: I see others from DASH-IF here, please jump in to clarify
> anything, if needed.
>
> Background
>
> Charles: There's an interop group at DASH-IF, DAInty: APIs for
> interactivity.
> .... A client could be implemented as a JavaScript library in a web page,
> or a streaming client in a particular device platform.
> .... The origin of this work is a related work item at 3GPP on service
> interactivity.
> .... The goal is to allow users to engage with auxiliary content alongside
> live or time-shifted content delivered over 3GPP, either broadcast or
> unicast network.
> .... DAInty provides hooks for interactivity, and also measurement on the
> use of interactivity.
> .... There's a work item description document and a technical report.
> .... DASH-IF was requested by 3GPP to support API specification.
>
> Architecture
>
> Charles: The APIs are between a DASH client and a service application. The
> interactivity events could be delivered over broadcast or unicast HTTP.
> .... The 3GPP's focus is more on the broadcast case. Event data and media
> data is passed over the 3GPP network to the interactive application.
> .... Event-type information is handled by the DASH client.
> .... At 3GPP there's interest in leveraging the DASH client to record
> measurement of interactivity usage.
> .... It's possible for the application to provide this directly, but 3GPP
> already has a reporting framework, so we want to use this for application
> level events.
>
> APIs
>
> Charles: There are 3 APIs to be developed. The first one allows an
> application to subscribe to DASH stream messages, which carry interactivity
> information.
> .... This could be in-band (emsg) or MPD events.
> .... An application subscribes to a scheme id and value, and the client
> dispatches matching events to the application.
> .... The second API allows an application subscription to an ISO BMFF
> timed metadata track.
> .... The third is interactivity usage measurement reporting. The usage
> measurement is delivered to the DASH client, which delivers it to the
> network server.
>
> DASH processing model
>
> Charles: The timeline diagram shows an in-band event message, carrying
> some app specific information.
> .... It's been subscribed to by some application.
> .... An emsg box, which is defined to be at the beginning of a segment.
> .... The payload has a presentation time for the start of the event.
> .... The duration could span multiple segments.
> .... The emsg event must arrive no later than the beginning segment to
> which it applies, for the event start and duration to be fully usable.
> .... The lower picture shows "event dispatch mode", passing from the DASH
> client to the application.
> .... Two cases are possible: dispatch at the time the event arrives (mode
> 1), or dispatch at the event start time (mode 2).
> .... The application can register with the DASH client which mode is
> preferred.
> .... Mode 1 gives the application more time, but in either case the
> application should be able to respond in time.
>
> Work status
>
> Charles: The DAInty APIs are intended both for web apps, and native apps
> on a particular device platform, as there are both application environments
> in the 3GPP model.
> .... For the web app case, the desire is for interactivity apps to be
> defined in DASH.js, an open source DASH player supported by DASH-IF.
> .... There are lots of other clients available, but the one developed
> under the auspices of the DASH-IF is DASH.js.
> .... There is already support in DASH.js for SCTE-35, but we want support
> for more generic events.
> .... We're writing in IDL, so not implementation specific. This allows the
> IDL to be mapped to a specific language or platform.
> .... The present focus of work is on the DASH event API, then later on the
> timed metadata track and interactivity usage reporting.
> .... Does W3C see value in standardising an IDL description? Could use
> WebIDL.
> .... We have an agressive timeline, aligned with 3GPP, to complete by end
> of the year.
> .... We're well on the way with the API definition.
>
> Questions and discussion
>
> Kaz: Is it OK to distribute the slides to the M&E IG public mailing list?
>
> Giri: Yes, I will share them after the call.
>
> Kaz: You're working on a concrete API defintion in IDL. Are you interested
> not only in WebIDL but also TypeScript? Other W3C groups have been
> struggling to define event APIs in WebIDL.
>
> Charles: We'd be happy to look into that, if W3C has some guidance.
> .... We also looked at OMG IDL. Could you provide some more information?
> .... We can pick up offline with Giri.
>
> Giri: We did talk about the choice of IDL. The one experience I had was in
> Fido Alliance, where we used it to define native APIs.
> .... I asked Charles to present on this, it's because of a percieved need
> that the web standards have not met.
> .... We want to feed this into WICG.
>
> Charles: I'm not sure how much WebIDL expertise we have in DASH-IF.
>
> Giri: I'd recommend that when this work is in a sufficient state, DASH-IF
> should send this to W3C under liaison.
>
> Will: IDL isn't our strength, certianly, but we will send under liaison.
>
> Francois: I'm trying to understand how the event model differs from the
> usual Web eventing model.
> .... I understand your mode 2 is similar to the media element's event
> model, but mode 1 differs.
>
> Charles: Yes, emsg events could arrive at the beginning of a segment. The
> emsg box describes the start time and duration
> .... Notifying the application when the event is received gives it
> additional time.
>
> Will: I can confirm that. We have a preference for mode 1, so that we can
> put the events into a helper library to dispatch at the right time.
> .... It's never wrong to get an event early, but it is wrong to get it
> late.
> .... We've had a long thread on this. My concern with the document is with
> some of the language, such as event API.
> .... I want to make sure that what's proposed fits, it is specific to
> certain players?
> .... Need to avoid creating a confusing API surface for developers.
>
> Charles: 3GPP's interest is to have some reference implementation, could
> be implemented in DASH.js
>
> Will: another comment on language: interactivity is a broad term. Web
> developers think of click events on UI controls. We need a clear definition.
>
> Charles: Absolutely.
>
> <tidoust> scribenick: tidoust
>
> Chris: You say the event carries a start time and a duration. Is it
> possible for an event to be unbounded?
> .... So you don't know when it ends in the future.
>
> Charles: In principle, it is. We're basing that on the DASH specification.
> I don't remember exactly what's in there.
>
> Giri: Internally, we've struggled with TTML on ways to define an unbounded
> metadata cue. "Display once, keep it on the display until I tell you to get
> rid of it". We haven't found something in TextTrack that lets us do that
> out of the box.
>
> Chris: OK, I would have to go back and read the DASH spec.
> .... My next question is related to the work we're doing in this Task
> Force. We're producing a use cases & requirements document to gather
> requirements for user agents that would support these kind of events.
>
> <kaz> Media Timed Events Use Cases and Requirements document:
> https://w3c.github.io/me-media-timed-events/
>
> Chris: We have agreement on generic use cases, I believe, but need more
> details on requirements that you may bring here.
> .... For example, does the application have to specify the scheme id and
> value to receive events?
>
> Charles: There may be many types of events and the application may want to
> know about specific events that it's interested in, so yes, there's a way
> to subscribe to events based on the scheme ID URL.
>
> Chris: I think the HbbTV specification has a similar API for exposing
> events that operates in a same way.
>
> Charles: Is this the part that transfers the event to some TextTrack?
>
> Chris: Yes, that's right.
> .... In HbbTV, it's always an opt-in from an application perspective.
> .... And then the event itself is defined as a DataCue that exposes the
> binary data from the event message.
> .... Is that what you're looking for as well, a generic blob that the
> application knows how to interpret?
>
> Charles: [looking up examples of events in the DASH spec]
> .... Looking at the DASHEventMessageBox definition, the intent is to
> provide the timing information as well as the data to the application so
> that the application can make use of it.
>
> Chris: Related to MPD events. Because they're delivered through the MPD,
> you need an API that allows the application to pass the events into the
> media player?
>
> Charles: Yes, not explained here, but we have a set of slides on that as
> well.
> .... Both types should be supported.
>
> Chris: I guess it's just that there would need to be an API to allow an
> application to pass in objects with start time and end time and data,
> rather than having that extracted from the media segments by client.
>
> Charles: Yes, the DASH client extracts that info, the only difference is
> where the info resides.
>
> Chris: I'm thinking of this from a Web platform API point of view, we
> would want the user agent to take care of the emsg box parsing, whereas
> it's application-level JavaScript that would parse the events embedded in
> the MPD. The requirement is the same, but done by different things.
> .... Another question: If we were to push for a Web platform API to do
> this, is the preference still for the Mode 1? As I understand it, the
> current model that we have, using TextTracks, is closer to Mode 2.
>
> Charles: I don't know which one is the ideal mode. We would want to allow
> both modes, and allow the client to subscribe to the mode it's interested
> in..
> .... I don't know that we want to specify one or the other.
>
> <cpn> scribenick: cpn
>
> Francois: There's no event when a change is made to the list of cues, even
> when not active.
> .... Maybe it's just a matter of adding an event when the list changes.
>
> <tidoust> scribenick: tidoust
>
> Chris: Interesting that we don't have something for mode 1 on the Web.
>
> <cpn> scribenick: cpn
>
> Will: I'd support Francois on that.
>
> <tidoust> scribenick: tidoust
>
> Chris: We shouldn't be going too deep into API design, but what you've
> been describing Charles really helps shape the specific API requirements.
>
> Timing and synchronization
>
> <kaz> https://github.com/w3c/media-and-entertainment/issues/4
>
> Chris: What kind of timing requirements did you specify for event
> dispatch? For example, Mode 2 event dispatch is probably more timing
> sensitive.
>
> Charles: I think the idea is that the application developer should be
> cognizant of the fact that there may be some sort of delay.
> .... It probably cannot be done with guarantees within the right
> millisecond..
> .... It would be important that the DASH player provides the current
> presentation time when it performs the dispatch.
> .... This allows synchronization.
>
> Giri: We're almost out of time for this call. We can continue the
> discussion offline.
> .... Thank you, Charles, for this presentation.
>
> <kaz> [adjourned]
>
> Summary of Action Items
> Summary of Resolutions
> [End of minutes]
> Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
> $Date: 2018/08/21 08:33:41 $
>
>
> -----------------------------
> http://www.bbc.co.uk
> This e-mail (and any attachments) is confidential and
> may contain personal views which are not the views of the BBC unless
> specifically stated.
> If you have received it in
> error, please delete it from your system.
> Do not use, copy or disclose the
> information in any way nor act in reliance on it and notify the sender
> immediately.
> Please note that the BBC monitors e-mails
> sent or received.
> Further communication will signify your consent to
> this.
> -----------------------------
>
>
>