Minutes from Media Timed Events Task Force call, 15 April 2019

Dear all,

The minutes from Monday's Media Timed Events Task Force call are available [1], and copied below.

Kind regards,

Chris (Co-chair, W3C Media & Entertainment Interest Group)

[1] http://www.w3.org/2019/04/15-me-minutes.html

W3C

- DRAFT -

Media and Entertainment IG - Media Timed Events TF

17 Apr 2019

Agenda

Attendees

Present
	Kaz_Ashimura, Chris_Needham, Song_Xu, Scott_Low, Will_Law, Vladmir_Levantovsky, Rob_Smith, Mark_Vickers
Regrets
	Nigel_Megitt
Chair
	Chris
Scribe
	Kaz

Contents

Topics

1.Status of IG note
2.W3M feedback
3.Time marches on
4.Defining 'event'
5.Events with unknown end time
6.WICG progress
7.Next call

Summary of Action Items
Summary of Resolutions

<scribe> Scribe: kaz

# Status of IG note

Chris: Here's a link to the document

<cpn> https://w3c.github.io/me-media-timed-events/

Chris: It's more or less ready for publication.
 ... As a brief overview, it talks about two things.
 ... Firstly, a set of use cases driving the need for some kind of API for metadata events,
 ... for information that is synchronized with media, and events embedded with the media stream.
 ... These could be in-band and out-band. Leads to the need to standardize existing DataCue API.
 ... The second is timing requirements, where we want web applications to handle events in a timely manner.
 ... For example, Nigel mentioned need for more accurate event timing for text track cues, to more closely align caption rendering and the video content.
 ... Since text track cues and data cues use the same underlying mechanism, makes sense to tackle both together.
 ... If you have any additional feedback, more than happy to include it.

# W3M feedback

Chris: Kaz mentioned we got some feedback from W3C Management to add some text to explain our future intent for this Note.

Kaz: Right.

Chris: What I propose for this issue is to add some text to explain that we plan to take the DataCue part of this to WICG,
 ... for incubation, and to take the synchronization issues to WHATWG, as this involves potential changes to the
 ... "time marches on" algorithm, part of HTML, for handling text track cues, and how events are triggered during media playback.
 ... Any questions?

# Time marches on

Will: Why the need to change time marches on, I'd expect the logic there to be consistent?

Chris: This is a good point. I don't think we're proposing any logic changes,
 ... it's a fairly complicated algorithm, so would be hesitant to change it..
 ... I think the only change we need is how frequently it runs.
 ... If you read the spec, right now it's only guaranteed to run every 250ms.
 ... Nigel did some tests, showed in practice that Chrome runs at about 250ms, but Firefox runs it much more frequently.

Will: Is there thinking towards an API for applications to set the frequency? What about sleep mode. Chrome behaves differently when a tab is not active, the timing could be up to about a second or so. Would this be subject to the same sleep mode limitations?

Chris: We haven't discussed this so far, those are really good questions.
 ... There are things here which have an impact on battery life, e.g., on mobile devices.
 ... What I'd like to do is have the conversation with implementers. We could clarify our requirements if needed.
 ... I'm not sure if the timing requirements are the same for inactive tabs, interested in hearing feedback on that.

Will: I'm not sure.

Chris: Should I capture this point as a GH issue?
 ... For the IG Note, I want to capture use cases and requirements.

Will: So this would be a requirement from my viewpoint, for a user settable period.
 ... But if the device can't honour it, there needs to be a way to tell the app, e.g., you asked for 33ms but you can't have that.
 ... If it goes to sleep, there's a sleep event which implies a change in update rate, or a new event telling you what the new update rate is.
 ... I don't know if those are feasible, but having some way for the app to know what the timing resolution is, is useful.

Scott: What's the requirement for the app to be able to define it, versus the browser respecting a high preceision, specified timer loop?

Will: A couple of use cases. One is motion tracking, where you have cursor above a basketball player's head, which should stay there.
 ... So you want updates every few frames to update the coordinates. This needs fast updates.
 ... But if you know what is changing is only every 5 minutes, then updates every few frames aren't necessary.
 ... I'm thinking you could tune the update cycle to match the precision you need.

Scott: Makes sense.

Will: Browser implementers may argue they want to do it only one way, and may not want to expose this to users.

Scott: This is worth capturing, but I think there's an aversion to allowing apps to be able to specify timing, because of things like battery life.

Chris: It's worth capturing in GitHub, so we can track this. Not sure whether it should be go into the IG note.

Scott: Sounds like a reasonable approach.
 
<scribe> ACTION: Chris to add GitHub issue describing application control over update frequency.

# Defining 'event'

<cpn> https://github.com/w3c/me-media-timed-events/issues/41

Chris: Cyril raised an issue saying that event is not defined.
 ... It's true we don't clearly define what "event" is.
 ... I think we can resolve it, as we are talking about events in the HTML / DOM sense.
 ... HTML spec talks about cues, and each cue has onenter and onexit events associated,
 ... and it's the synchronization of these events that we're interested in.
 ... The term "event" is also defined in the MPEG world, as emsg.
 ... He also asks "is an event instantaneous or long-running?", made me ask whether it's possible to create a TextTrackCue
 ... with zero duration, i.e., the start and end time are the same, or is there some minimum duration applied?
 ... I think we can safely state what we are talking about is DOM events, but clarifying the terminology would help.
 ... Cue is established terminology in HTML.
 
<scribe> ACTION: Chris to reply to issue 41 and clarify event terminology.

# Events with unknown end time

<cpn> https://github.com/w3c/me-media-timed-events/issues/39

Chris: Rob, you have a requirement to set the endTime as not known, so it applies to the end of the media.

Rob: This is a common use case for WebVMT, which is annotating a map, a lot of annotations persist to the end of the media.
 ... For a live stream, the end of the media is not known, because it's stream, but we want the annotations to remain until the end of the media.

Chris: Do you think the text within the Note suffice, can we close this issue??

Rob: I think it does. I have discussed with Apple who said there's a way to handle an unknown end time, using Infinity.
 ... That was more for the media duration, rather than cue end times, but the same mechanism could be used.
 ... The spec doesn't say it explicitly, but wouldn't a big modification, rather a clarification.
 ... Also the use cases Nigel raised, so this may be part of a larger problem.
 ... There may be two separate requirements: one for an unknown end time, to the end of the media,
 ... another for an end of chapter or end of event which is interactive in some way.

Chris: Is this related to having ability of specifying endTime?

Rob: Yes, changing it after the start of the cue.

Chris: I don't think we have wording which captures this point precisely.
 
<scribe> ACTION: Rob to review the IG note for cue end time.

Chris: Don't need to go through the other open issues.

Mark: Regarding the last issue, that Infinity would be a minor modification, is that against the HTML spec?

Chris: Yes, as that's where all the text track stuff is defined.

Mark: Maybe we should raise an issue for the HTML spec itself, can follow this up in parallel with the IG note.

Rob: I think it's a change, so not just a clarification as it's not currently supported.

Mark: right, I understand it's not just a clarification. If it's a straightforward change, can we file this separately?

Chris: I think we could do that. We have a couple of issues to raise with HTML, one is about time marches on, the other for
 ... There is no reason we can't propose an issue for HTML now.

Mark: I totally agree, the HTML issue could reference this document, which reinforces it.

# WICG progress

Chris: We now have a repo in WICG

<cpn> https://github.com/WICG/datacue

Chris: This was maybe one month ago or so, through contact with the WICG chairs.
 ... The repo contains an explainer for DataCue, based on the IG note, doesn't include the timing stuff.

<cpn> https://github.com/WICG/datacue/blob/master/explainer.md

Chris: It's really a copy of main use cases from the IG Note,
 ... with an intro that Giri, Mark and I put together for the Discourse forum.
 ... Because this is an explainer in WICG, we can talk about API design, so there's some example code
 ... which is based on the Webkit DataCue implementation.
 ... I'm hoping this is now the place we can have discussion, as part of the WICG rather than the M&E IG.
 ... If you look through the document, there are some TODO comments, relating to adding example code to show how the API works.
 ... It's critical to have implementer support, so I'm a bit hesitant to go into too much detail without their involvement.
 ... I have some feedback from Apple, and I know other browser vendors have been supportive.
 ... Some of them have not given response yet. Advice on moving things forward is welcome.

Rob: At TPAC in Lyon, the Spatial Data on the Web group had several people from Google present.
 ... They were interested in DataCues and the idea behind WebVMT.
 ... I have a couple of contacts, if you need them.

Chris: I can contact them again as a next step.

Mark: Is the process to have the discussion in the WICG Discourse, or is that only for before you get a repo?

Chris: I think discussion can happen there at any time, having more voices could be helpful.
 ... This proposal is a media industry thing, not just me.
 ... But I think the Discourse is more for the initial discussion, once you have a repo you can use GitHub issues.
 ... The other thing I can do is email privately among browser implementers..
 ... Not sure whether the media teams of browser implementers are in touch with each other about this.

Mark: I think private email thread would be OK, I'd be happy to join that.

Chris: That sounds good, I'll do that, with the IG co-chairs included.
 
<scribe> ACTION: Chris to email browser implementers for feedback on DataCue.

Chris: Thinking a few steps ahead, if there are other people we need involve,
 ... for example which in-band events do we have consensus to support?
 ... emsg is particularly important. From the information that Apple shared, Webkit has support for other in-band event types, so we need to discuss which events are to be standardised.
 ... Then there's the question of specifying how to extract the information from media containers.
 ... We'll need somebody familiar with media containers.

Mark: I would suggest that, the issue everybody is interested is CMAF.
 ... There's a commitment to align on that, and a lot of interest in that.
 ... Pushing emsg and CMAF should be the leading discussion point.
 ... emsg is a required part of CMAF, and so there's a gap in the web platform for full CMAF support.
 ... Regarding authoring, would be best if someone from a browser company could do that.

Chris: I hope to discuss this next, it need people who're familiar with implementations.

Rob: I have no problem with that suggestion, but also want to see WebVMT in there,
 ... as it's a different angle on the metadata, which is very web specific.
 ... WebVMT has indexing and search aspects that are less of interest to you in media, but wouldn't want those to be missed.

Chris: I'm happy to include this.
 ... What do we want to say about the search use case, for example?

Rob: I put together a couple of use cases last year, search wasn't included in the IG note.
 ... Concern about it being overlooked.
 ... Perhaps we could capture it in the explainer. There's a placeholder for unknown cue end time.

Chris: This is different use case from the search case?

Rob: Right

Chris: I thought the search would be handled by a web service. The same data can be rendered as WebVMT for playback in the browser via DataCue.

Rob: The cues would need some associated data to say this is the result of the search, that it matches in a particular way.
 ... If it's a wider search, so it needs to indicate the agreement it has with the search.

Chris: Could this be handled by the data payload we give to DataCue?
 ... Maybe we could have detailed discussion offline. I don't want to miss an important use case, but I may need to understand better.

Rob: Sure.
 
<scribe> ACTION: Chris and Rob to discuss the indexing and search use cases, and possibly add to the IG note or explainer.

Mark: To clarify, I'm suggesting CMAF and emsg as a point of urgency, but all the use cases are still included,
 ... as a way to move from the requirements phase to the spec writing phase..

Rob: OK.

Chris: Within the media industry there is a push for CMAF, which is a driver for moving this forward.
 ... I expect implementers will want to see broad applicability, so there are more use cases.
 ... We should be trying to bring both aspects.
 ... Any further comments before we close the call?

(none)

# Next call

Chris: I'd like to continue having calls as we make progress.
 ... Scheduled for third Monday of every month.
 ... Next call is May 20, need to check for possible overlap with the Fraunhofer Media Web Symposium.
 ... We're planning a session to talk about media Web APIs there, including DataCue.

(May 21-22 FOKUS Media Web Symposium)

Chris: I'll announce details, depending on availability.

[adjourned]

Summary of Action Items

[NEW] ACTION: Chris and Rob to discuss the indexing and search use cases, and possibly add to the IG note or explainer.
[NEW] ACTION: Chris to add GitHub issue describing application control over update frequency.
[NEW] ACTION: Chris to email browser implementers for feedback on DataCue.
[NEW] ACTION: Chris to reply to issue 41 and clarify event terminology.
[NEW] ACTION: Rob to review the IG note for cue end time.
  

Summary of Resolutions

[End of minutes]


Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)

$Date: 2019/04/17 08:32:35 $

Received on Wednesday, 17 April 2019 09:36:47 UTC