RE: MPEG-2 TS metadata cue guidelines

From: Bob Lund [mailto:B.Lund@CableLabs.com]
Sent: Monday, March 24, 2014 1:38 PM
To: Clift, Graham; Brendan Long; public-inbandtracks@w3.org
Subject: Re: MPEG-2 TS metadata cue guidelines



From: <Clift>, Graham <Graham.Clift@am.sony.com<mailto:Graham.Clift@am.sony.com>>
Date: Monday, March 24, 2014 at 10:50 AM
To: Bob Lund <b.lund@cablelabs.com<mailto:b.lund@cablelabs.com>>, Brendan Long <B.Long@cablelabs.com<mailto:B.Long@cablelabs.com>>, "public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>" <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Subject: RE: MPEG-2 TS metadata cue guidelines



From: Bob Lund [mailto:B.Lund@CableLabs.com]
Sent: Monday, March 24, 2014 9:45 AM
To: Clift, Graham; Brendan Long; public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>
Subject: Re: MPEG-2 TS metadata cue guidelines



From: <Clift>, Graham <Graham.Clift@am.sony.com<mailto:Graham.Clift@am.sony.com>>
Date: Monday, March 24, 2014 at 10:35 AM
To: Bob Lund <b.lund@cablelabs.com<mailto:b.lund@cablelabs.com>>, Brendan Long <B.Long@cablelabs.com<mailto:B.Long@cablelabs.com>>, "public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>" <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Subject: RE: MPEG-2 TS metadata cue guidelines



From: Bob Lund [mailto:B.Lund@CableLabs.com]
Sent: Monday, March 24, 2014 9:27 AM
To: Clift, Graham; Brendan Long; public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>
Subject: Re: MPEG-2 TS metadata cue guidelines



From: <Clift>, Graham <Graham.Clift@am.sony.com<mailto:Graham.Clift@am.sony.com>>
Date: Monday, March 24, 2014 at 10:11 AM
To: Brendan Long <B.Long@cablelabs.com<mailto:B.Long@cablelabs.com>>, Bob Lund <b.lund@cablelabs.com<mailto:b.lund@cablelabs.com>>, "public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>" <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Subject: RE: MPEG-2 TS metadata cue guidelines

I have one suggestion. Instead of playing around with arbitrary endtimes  why not set the starttime to the currentTime of playback when the cue is created (making it immediately active when created) and set the endtime to the media resource time

What is 'media resource time'?
[<Graham>] Well that is terminology you use in the mapping spec. It is the time of the metadata with respect to the MPEG-2 TS timeline (eg PTS) clock. The startTime in contrast would be the HTML5 Media Element currentTime when the cue is created in UA, which is equivalent to
                startTime = endTime - (UA decoder buffering time) + UA cue generation processing time

Why not endTime = startTime + 250msec? Seems easier.
[<Graham>]
I explained the reasoning but I will reiterate. 250msec is just some arbitrary number. Rather than an arbitrary number why not just have the startTime before the mediaresource time to allow the application to prepare for the cue onexit event?

Is the following another way of stating your proposal?

Let PTS0 be the the MPEG-2 TS PTS corresponding to video.currentTime = 0.
Let PTS1 be the MPEG-2 TS PTS of the video stream when the UA is ready to create a metadata text track cue from an associated elementary stream. The UA sets cue.startTime = video.currentTime and cue.endTime = max(250msec, PTS1 - PTS0).


[<Graham>] Yes that is want I mean, and I think it will give a consistent behavior with good synchronization. In addition it solves other issues that arise from arbitrary endTimes like availability of non activeCues for live content where seekability is zero. (i.e the previous suggested approach of setting the startTime to currentTime and endTime to currentTime+250ms might mean that the cue is no longer active when the currentTime matches the real time of insertion point. Such 'live content' inactive cues might then be automatically deleted by the UA before they were expected to be utilized according to party responsible for encoding the cues.)

Descriptively the approach I am suggesting could be written as:

DataCue<http://www.w3.org/html/wg/drafts/html/master/single-page.html#datacue> startTime<http://www.w3.org/html/wg/drafts/html/master/single-page.html#dom-texttrackcue-starttime> is set to the media element<http://www.w3.org/html/wg/drafts/html/master/single-page.html#media-element> currentTime<http://www.w3.org/html/wg/drafts/html/master/single-page.html#dom-media-currenttime> at the point in that the user agent creates the cue on the TextTrack<http://www.w3.org/html/wg/drafts/html/master/single-page.html#texttrack>. The application may use a cuehange<http://www.w3.org/html/wg/drafts/html/master/single-page.html#event-media-cuechange> event on the TextTrack<http://www.w3.org/html/wg/drafts/html/master/single-page.html#texttrack> to begin pre-processing the cue<http://www.w3.org/html/wg/drafts/html/master/single-page.html#texttrackcue> (the application may wish to create an exit<http://www.w3.org/html/wg/drafts/html/master/single-page.html#event-media-exit> synchronization event, for instance). The enter<http://www.w3.org/html/wg/drafts/html/master/single-page.html#event-media-enter> event is not useful for DataCues<http://www.w3.org/html/wg/drafts/html/master/single-page.html#datacue>.

DataCue<http://www.w3.org/html/wg/drafts/html/master/single-page.html#datacue> endTime<http://www.w3.org/html/wg/drafts/html/master/single-page.html#dom-texttrackcue-endtime> is set to greater of the closest available previous value of the PTS clock extracted from the MPEG-2 TS video stream with respect to the metadata cue insertion point ( adjusted for any PTS clock offset with respect to media element<http://www.w3.org/html/wg/drafts/html/master/single-page.html#media-element> currentTime<http://www.w3.org/html/wg/drafts/html/master/single-page.html#dom-media-currenttime> equal to zero ) and startTime<http://www.w3.org/html/wg/drafts/html/master/single-page.html#dom-texttrackcue-starttime> plus 250ms. Once endTime<http://www.w3.org/html/wg/drafts/html/master/single-page.html#dom-texttrackcue-endtime> has passed and any exit<http://www.w3.org/html/wg/drafts/html/master/single-page.html#event-media-exit> event has fired the DataCue<http://www.w3.org/html/wg/drafts/html/master/single-page.html#datacue> becomes inactive, and in the case of live content, may be deleted from the list of cues. Firing of the exit<http://www.w3.org/html/wg/drafts/html/master/single-page.html#event-media-exit> event could be used to trigger a synchronized event associated with the DataCue<http://www.w3.org/html/wg/drafts/html/master/single-page.html#datacue>.







(making it inactive after this time). An application can use the onexit event to accurately time the DataCue event and onenter to prepare for this event like fetching a resource for an ad insertion. Of course we will want to specify a minimum endtime-starttime value (which could be zero)

Regards

Graham  Clift

From: Brendan Long [mailto:B.Long@cablelabs.com]
Sent: Monday, March 24, 2014 8:59 AM
To: Bob Lund; public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>
Subject: Re: MPEG-2 TS metadata cue guidelines

Oh never mind. What I suggested doesn't make sense with other metadata cues.

From: Bob Lund <B.Lund@CableLabs.com<mailto:B.Lund@CableLabs.com>>
Date: Monday, March 24, 2014 at 9:38 AM
To: Brendan Long <b.long@cablelabs.com<mailto:b.long@cablelabs.com>>, "public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>" <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Subject: Re: MPEG-2 TS metadata cue guidelines



From: Brendan Long <B.Long@cablelabs.com<mailto:B.Long@cablelabs.com>>
Date: Monday, March 24, 2014 at 9:10 AM
To: Bob Lund <b.lund@cablelabs.com<mailto:b.lund@cablelabs.com>>, "public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>" <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Subject: Re: MPEG-2 TS metadata cue guidelines

I'm not sure if it's any better, but another option would be to set it to very large number (infinity if we changed the spec to allow that) and then fix it once we know what the end time is. The advantages would be:
*         track.activeCues[0] should always be the current PMT.
There is no proposal in the submitted HTML5 bugs for PMT track cues, I'm only talking about metadata cues in general. What is the advantage of track.activeCues[0] being the last received cue?
*         The historical data in track.cues will tell you the actual start and end times for previous PMTs, but I'm not sure if that's useful practically.
The disadvantage is that you'd need to mess with cues after creating them.

If there is no advantage for having the last received cue in track.activeCues[0] then my proposal eliminates the UA having to reset times.


From: Bob Lund <B.Lund@CableLabs.com<mailto:B.Lund@CableLabs.com>>
Date: Monday, March 24, 2014 at 8:41 AM
To: "public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>" <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Subject: MPEG-2 TS metadata cue guidelines
Resent-From: <public-inbandtracks@w3.org<mailto:public-inbandtracks@w3.org>>
Resent-Date: Monday, March 24, 2014 at 8:42 AM

CG,

Bug 25005<https://www.w3.org/Bugs/Public/show_bug.cgi?id=25005> is about how to set endTime for MPEG-2 TS metadata text track cues. The issue is that there is no explicit endTime for these cues. I proposed<https://www.w3.org/Bugs/Public/show_bug.cgi?id=25005#c4> setting the endTime to the startTime. The HTML5 editor doesn't think this is a good idea<https://www.w3.org/Bugs/Public/show_bug.cgi?id=25005#c5>.

The reality is that it doesn't really matter since the application understands, and acts on, the cue. So, an alternate proposal is to set the endTime = startTime + minDelta. Where minDelta = the minimum time so that the cue entry and exit handlers will operate properly vis-a-vis the  "time marches on" algorithm. Comments?

Bob