Re: Media timed events use cases and gap analysis

Chris and all

Following up on Bob Lunds comment [1].

"... a DataCue chicken and egg problem. Browser vendors have been reluctant
to support DataCue because of lack of applications. Applications are
stymied due to the lack of the DataCue API."

I find this description of the situation easy to agree with. I also thought
I should mention that we have done quite a bit of research work on this
topic over recent years. In particular - in reference to the DataCue API
and the texttrack mechanism, we've published a paper [2] highlighting
several issues with the current situation (some of them already mentioned
in this thread), as well as an effective solution to all of them (at least
those issues we have been able to identify ;)). I think some of the ideas
expressed in this paper should be of relevance to this discussion.

Some key insights/demands:

- *stateful* - waiting for the next event is not fun - the mechanism must
always be able to give you the correct state immediately and whenever.
Also, it should not drop events etc.
- *maximize precision* - applications have varying demands with respect to
precision. A general mechanism should strive to be useful for as many
(future) applications as possible, therefore precision must be maximized
(sub 10 milliseconds is no problem today in JS, so no reason to aim for
something worse than that)
- *timeout-based* executions for scheduling of enter/exit events, as
opposed to stepwise evaluation (ref time-marches-on)
- make it work with any data - *regardless of data format *
- make it work with any data - *regardless of transport mechanism* (support
both in-band transport and/or independent transport)
- make it work with *dynamic* data sources

Finally, and perhaps most contrary to intuition:
- make it also work *without* a Media Player !


Best regards,

Ingar Arntzen, Chair Multi-device Timing CG

PS. If you are interested in reading the paper, but don't have access to
ACM, just send me a request for copy.

[1] https://lists.w3.org/Archives/Public/public-web-and-tv/2018Mar/0000.html
[2] https://dl.acm.org/citation.cfm?id=2910614




2018-03-01 17:19 GMT+01:00 Bob Lund <bob.lund@gmail.com>:

> Chris and All,
>
> Here are some thoughts on an approach to address the gaps identified in
> the "Synchronized Event Triggering" use case [1].
>
> As pointed out, the "Sourcing In-band Media Resource Tracks" document [2]
> needs to be updated to define the creation of metadata text tracks and cues
> when DASH events are found in the MPD or media containers of interest.
> Given this, there is the question of how to create browser implements.
>
> Many, if not most, browser adaptive bitrate implementations are JavaScript
> based that use media source extensions. With this, it is pretty
> straightforward to extend the JS player to extract the DASH events and
> create metadata text tracks and cues using the HTML5 text track API [3].
> Then there is the question of the cue API.
>
> Also as pointed out, the DataCue API would be a good way to expose these
> events to web applications but DataCue is not supported in browsers. This
> can be addressed by implementing a DataCue JS polyfill to create a DataCue
> API on top of VTTCue, which is widely supported in browsers.
>
> I've built a proof-of-concept of this in the past using the DASH.js player
> that worked across Chrome, Firefox and Safari at that time.
>
> The approach outlined above might address what seems to be a DataCue
> chicken and egg problem. Browser vendors have been reluctant to support
> DataCue because of lack of applications. Applications are stymied due to
> the lack of the DataCue API. With JS players and the polyfill, content
> providers can address the lack of browser support and implement timed media
> applications, thereby demonstrating a demand to browser vendors.
>
> Regards,
> Bob Lund
>
> [1] https://github.com/w3c/media-and-entertainment/blob/mast
> er/media-timed-events/use-cases-and-gap-analysis.md
> [2] https://dev.w3.org/html5/html-sourcing-inband-tracks/
> [3] https://www.w3.org/TR/html52/semantics-embedded-content.
> html#text-track-api
>
>> ------------------------------
>> *From:* Chris Needham <chris.needham@bbc.co.uk>
>> *Sent:* Friday, February 16, 2018 5:19 AM
>> *To:* public-web-and-tv@w3.org
>> *Subject:* Media timed events use cases and gap analysis
>>
>> Dear all,
>>
>> Following the discussion on media timed events during the last M&E IG
>> call [1], I have started a document to gather use cases and requirements
>> [2]..
>>
>> This is just an initial draft so far, intended to provide input to the
>> work on any spec changes needed.
>>
>> The document needs your input, so that it reflects the IG's requirements.
>> One particular area that needs attention is improved use case descriptions,
>> particularly from the point of view of end users or content authors, as
>> some of what's currently written is technology rather than use-case driven.
>>
>> Please submit feedback via the GitHub issue tracker (pull requests are
>> very welcome), or via this mailing list if you prefer.
>>
>> Kind regards,
>>
>> Chris (Co-chair, Media & Entertainment Interest Group)
>>
>> [1] https://www.w3.org/2018/02/01-me-minutes.html
>> [2] https://github.com/w3c/media-and-entertainment/blob/master/m
>> edia-timed-events/use-cases-and-gap-analysis.md
>>
>> <https://github.com/w3c/media-and-entertainment/blob/master/media-timed-events/use-cases-and-gap-analysis.md>
>> w3c/media-and-entertainment
>> <https://github.com/w3c/media-and-entertainment/blob/master/media-timed-events/use-cases-and-gap-analysis.md>
>> github.com
>> media-and-entertainment - Repository for the Media and Entertainment
>> Interest Group
>>
>>
>>
>>
>> -----------------------------
>> http://www.bbc.co.uk
>> This e-mail (and any attachments) is confidential and
>> may contain personal views which are not the views of the BBC unless
>> specifically stated.
>> If you have received it in
>> error, please delete it from your system.
>> Do not use, copy or disclose the
>> information in any way nor act in reliance on it and notify the sender
>> immediately.
>> Please note that the BBC monitors e-mails
>> sent or received.
>> Further communication will signify your consent to
>> this.
>> -----------------------------
>>
>>
>>
>