Minutes from Media Timed Events Task Force call 17 August 2020

Dear all,

The minutes from today's Media Timed Events / WICG DataCue API call are now available [1], and copied below.

There was one action item:

1. Rob and Chris to submit PR for unbounded cue end times

Kind regards,

Chris (Co-chair, W3C Media & Entertainment Interest Group)

[1] https://www.w3.org/2020/08/17-me-minutes.html


W3C
- DRAFT -
Media and Entertainment IG - MTE-TF
17 Aug 2020
Agenda

Attendees

Present
  Franco_Ghilardi, Kaz_Ashimura, Rob_Smith, Chris_Needham, Gary_Katsevman, Andreas_Tai, Iraj_Sodagar, Michael_Stattmann, Nigel_Megitt, Will_Law

Regrets

Chair
  Chris

Scribe
  cpn, kaz

Contents

Topics
  Introduction
  Unbounded cue durations
  Use cases and DASH-IF feedback
  Next call

Summary of Action Items
Summary of Resolutions

<cpn> scribenick: cpn

# Introduction

<kaz> Agenda: https://lists.w3.org/Archives/Public/public-web-and-tv/2020Aug/0004.html

Chris: Two topics: unbounded cues, and use cases and contextualizing DataCue

# Unbounded cue durations

Chris: https://github.com/whatwg/html/issues/5297
.... Thanks to Rob for writing up the analysis
.... Since then I've draft a proposed change to the HTML spec
.... Proposal to adjust the endTime on TextTrackCue to allow unrestricted values, enables +Infinity
.... Motivation is chapters with known start but unknown end
.... Usage in inband emsg DASH events can be signalled with unbounded duration
.... Proposed change enables us to express this via the TextTrackCue API
.... Let's review the text, then we'll create a PR to get input from browser implementers

Rob: I seem to remember there's a data type change to allow Infinite end time

Chris: Yes, I think we need to change the IDL to use unrestricted double, I'll add it
.... I'm not sure we needed to add a new concept for unbounded cues

Rob: It's useful to have a unambiguous statement that's linked in the spec. Unbounded streams are well understood, but unbounded cues are new, so I wanted to define it precisely.

Chris: Any other views? If not, I'll draft a PR on the proposed text
.... Assuming there's support, there's implementation work and updating WPT

Kaz: Is that a PR against WHATWG HTML?

Chris: Yes, that's right

<kaz> https://github.com/whatwg/html/pulls PRs for whatwg/html repo

# Use cases and DASH-IF feedback

Chris: Conversations here and in DASH-IF, closely related
.... Putting DataCue in context for service interactivity and synchronized rendering
.... Can we describe end to end what a synchronized interactive service would look like?

Rob: What's missing from DataCue?

Chris: I think it's the broader context for how DataCue is used in an application
.... We can review the 3GPP-based streaming use cases, initially brought to W3C when we started work on this

<cpn> https://www.3gpp.org/ftp/Specs/archive/26_series/26.953/26953-g00.zip Interactivity Support for 3GPP-Based Streaming and Download Services

<kaz> scribenick: kaz

Chris: What this talks about is a video streaming service delivered along with auxiliary data and user interactivity
.... 4.2.1.1 Use Case #1 is an example of an interactive application.
.... DataCue can be used to specify the starting point.
.... One of the things that's interesting from DASH-IF discussion is the data is closely integrated with the rendering pipeline.
.... In their case, the rendering pipeline for video is a bit different from the Web mechanism.
.... Can we describe how a Web-based rendering model is aligned with the DASH-IF model, as proposed by their Event TF?

<cpn> https://dashif.org/docs/EventTimedMetadataProcessing-v1.0.2.pdf DASH events and timed metadata tracks timing and processing model and client reference model

Chris: And also related to the work done in MPEG around having HTML tracks inside the media container.
.... This idea came from realization of emsg is not the most appropriate tool for conveying web content.
.... The player needs to extract the Web-based media and synchronize it.
.... The first use case here is a triggered media overlay.
.... This is relatively straightforward on the web today.
.... A web application hosted on a Web server, and DataCue is primarily related to the metadata that triggers the interactivity.
.... Use case #2 click for info.
.... Similar to #1, the user can click on the combination of buttons/links.
.... Use case #3 dynamic interactive Ads.
.... Question here arises around timing of resources, e.g., stream switching..
.... My assumption here is ad insertion to the main content and switching back, rather than a picture-in-picture display.
.... Today, this can be done with multiple video elements.
.... One which is off-screen and prebuffered.
.... The web application shows this at the point of signalled cue.
.... This may have some limitations due to codec availability.
.... Related work in MEIG on Media Integration Guidelines.
.... Also using Media Capability APIs to query for seamless switching capability.
.... How exactly is the DataCue used by the web app.
.... Instead of multiple video elements, can we schedule stream switches using MSE buffers?
.... If anybody has any thoughts on exactly how this works in practice, would like to hear about that.
.... Use case #4 dynamic and personalized interactive Ads.
.... This is a personalized variation of UC #3.
.... It could be a local decision for which content to show using the user's device
.... and the user's contextual information.
.... I'll leave Use case #5, I've not considered this one yet.

Rob: As I see it, DataCue is the data delivery mechanism,.
.... Two separate issues, DataCue carrying the data, and latency.

Chris: Yes, and I think overall synchronized presentation is related to synchronized rendering.
.... Web Codecs is one possibility, where the web application controls presenting the video frames,
.... so other content can be closely synchronized.
.... One downside of Web Codecs is possible problem with EME-protected content.
.... For that you'd need to use existing mechanisms to synchronize the document overlay.
.... JS execution to be synchronized, but how closely synchronized?
.... We have put some requirements for HTML5 on triggered cue timing accuracy.
.... The proposal we have with the DASH-IF group is to work on a document that describes
.... DataCue with rendering for synchronized interactive use cases.
.... We need to get more voices to input for DataCue, to support the feature for browser implementations.
.... I feel what I need is help putting this together, help from you all is very welcome.

Kaz: Do you mean we would like to send our requirements to the DASH-IF group to improve their requirements document?

Chris: It's more about working collaboratively with them to produce a document.
.... We can use the WICG GitHub repo to host it, and coordinate the work through this group and the DASH-IF Event TF.
.... Does anybody have thoughts?

Rob: I have a use case, taking video with timed metadata and location,
.... about particular trajectory, e.g., vehicle moving around.
.... The DataCue provides the terrain information,
.... along with inband realtime use case,
.... potentially these two things could be aligned in more synchronized way.

Kaz: Implies possible collaboration between video games and physical devices as AR

Rob: Yes. The output overlay can be rendered in AR-manner.
.... DataCue in that case is a delivery mechanism.
.... And I'm happy to help with the use cases.

Chris: Thank you! Let's follow this up.
.... By the way, I have just got a message regarding Streaming Video Alliance use cases,
.... seems quite positive.
.... The goal here is to have enough supporting material for a discussion with Media WG at TPAC.

Kaz: Do we want to invite 3GPP guys and DASH-IF guys to our TPAC session?

Chris: Yeah, would like to do so if possible.
.... If you have any thoughts on the use cases, etc, please let me know.

# Next call

Kaz: September 21?

Chris: yes

[adjourned]

Summary of Action Items
Summary of Resolutions
[End of minutes]
Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2020/08/18 02:49:35 $

Received on Tuesday, 18 August 2020 10:54:32 UTC