RE: [MSE] Establishing the Presentation Start Timestamp

Hi Aaron,

  Yes this makes sense to me thank you for the response.

-K

From: Aaron Colwell [mailto:acolwell@google.com]
Sent: Wednesday, August 08, 2012 2:07 PM
To: Kevin Streeter
Cc: <public-html-media@w3.org>
Subject: Re: [MSE] Establishing the Presentation Start Timestamp

Hi Kevin,

I just wanted to circle back with you to make sure my answer was satisfactory. If so then I'll start working on updating the spec so I can resolve Bug 18389<https://www.w3.org/Bugs/Public/show_bug.cgi?id=18389>.

Aaron

On Tue, Jul 24, 2012 at 4:42 PM, Aaron Colwell <acolwell@google.com<mailto:acolwell@google.com>> wrote:
Using reply all this time... :)

---------- Forwarded message ----------
From: Aaron Colwell <acolwell@google.com<mailto:acolwell@google.com>>
Date: Tue, Jul 24, 2012 at 4:40 PM
Subject: Re: [MSE] Establishing the Presentation Start Timestamp
To: Kevin Streeter <kstreete@adobe.com<mailto:kstreete@adobe.com>>


Hi Kevin,

Yes. The app can either set an offset or just append and then 'videoTag.currentTime = videoTag.buffered.start(0)' which should contain the start time of the first segment.

Aaron

On Tue, Jul 24, 2012 at 4:37 PM, Kevin Streeter <kstreete@adobe.com<mailto:kstreete@adobe.com>> wrote:
Aaron, Mark,

  I'm a little unclear on how things will work for a live stream.  The user will typically start playback at some non-zero time, which represents the "live" end of the stream.  How does the timestamp offset account for this?  Does it requiring setting a negative offset that re-bases the stream to 0 so that playback begins immediately?

-K

From: Mark Watson [mailto:watsonm@netflix.com<mailto:watsonm@netflix.com>]
Sent: Tuesday, July 24, 2012 2:19 PM
To: Aaron Colwell
Cc: <public-html-media@w3.org<mailto:public-html-media@w3.org>>
Subject: Re: [MSE] Establishing the Presentation Start Timestamp


On Jul 24, 2012, at 2:13 PM, Aaron Colwell wrote:

Hi Mark,

Thanks for your comments. I too am starting to believe that we should just have the SourceBuffer timelines start at 0 and NOT derive the presentation start timestamp from the first segment appended. I agree that the timestamp offset mechanism should be used to handle any content that doesn't already start at 0. This might make things a little annoying for live streams, but it is a one time operation at the beginning of playback to figure out what the appropriate timestamp offset needs to be. I think the 90-99% case will be with content that starts at 0 so we should optimize for that. If offsets need to be applied the app needs to know about them or "discover" them by appending a segment to a "scratch" SourceBuffer and see what SourceBuffer.buffered reports.

Right - and if we need to provide a "cleaner" way for the app to peek the media timestamp we can do that later when the need is clearer.


I do have a question about your first append at 10 minutes example. Are you saying that you want the HTMLMediaElement to implicitly seek to the the start of the first media segment appended? I think it might be less surprising if default playback start position<http://dev.w3.org/html5/spec/media-elements.html#default-playback-start-position> stays 0 and the app manually sets HTMLMedaElement.currentTime to HTMLMediaElement.buffered.start(0) or the desired seek time if you know it.

Agreed - that's what I intended (even if that's not what I wrote ;-)

Otherwise, I'm not sure how to make the behavior you desire fits into the descriptions specified in the offsets into the media resource<http://dev.w3.org/html5/spec/media-elements.html#offsets-into-the-media-resource> section of the HTML spec.

Aaron

On Wed, Jul 18, 2012 at 11:55 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:

On Jul 18, 2012, at 11:03 AM, Aaron Colwell wrote:

Hi Mark,

Comments inline...

On Thu, Jul 12, 2012 at 2:32 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:



4. How close do the starting timestamps on the first media segments from each SourceBuffer need to be?
  - In this example I've shown them to be only 30 milliseconds apart, but would 0.5 seconds be acceptable? Would 2 seconds?
  - How much time do we allow here before we consider there to be missing data and playback can't start?
  - What happens if the gap is too large?

I think this is roughly the same question as 'what happens if I append a video segment which starts X ms after the end of the last video segment' ?

if X <= one frame interval, this is definitely not a 'gap' and playback continues smoothly. If X > 1 second this is definitely a gap and playback should stall (in the same way as it does today on a network outage).

For X values in between, I am not sure: implementations have to draw a line somewhere. A gap of multiple frame intervals could occur when switching frame rate. You might also get a couple of frame intervals gap when switching if you do wacky things with frame reordering around segment boundaries.

When looking at differences between audio and video, we need to be tolerant of differences as much as the larger of the audio frame size and the video frame interval.

if the gap is too large, this element just stays in the same state. Perhaps I append video from 0s and audio from 2s and this is because my network requests got re-ordered and any millisecond now I am going to append the 0-2s audio. Playback should start when that 0-2s is appended.


[acolwell] I agree. We need to come up with some spec text for this and then we can then debate the merits of these various magic numbers. Care to volunteer for this? :)

Ok, assign me a bug.


Any insights or suggestions would be greatly appreciated.

We have the same problem with push/popTimeOffset. Suppose I want your media above to appear at offset 200s in both audio and video source buffers. What I really want is for the audio to start at 200s and the video at 200.030ms.

In this case the application knows better than the media what the internal media times are. I know that the video segment has all the video from time 0s, even though the first frame is at 30ms. I really want to provide the actual offset to be applied to the internal timestamps, rather than providing the source buffer time that the next segment should start at.

[acolwel] One way I think we could get around this is to mandate that the media segments actually have a start time of 0. In WebM there is a Cluster timestamp and then all blocks are relative to this timestamp. If the Cluster timestamp is 0 and the first frame in the cluster is at 30ms then there is enough information for the UA to "do the right thing". I'm not sure if a similar mechanism exists in ISO.

Not really - the rather complex combination of decode times, composition offsets and edit lists results in a presentation timestamp for each sample on a global timeline (shared across all bitrates etc.). But if the timestamp of the first sample is X there is nothing to say, for example, "there are no other samples between time Y (< X) and X".

The application that creates the demuxed files just need to make sure the separate files both have the same segment start time.

That's not always possible because of skew caused by audio frame durations being different from video frame intervals.

I think we need to say that all source buffers share a common global timeline and that timestamps in the media segments must be mapped to that in a way that is common across source buffers. This means any offset applied to media internal timestamps needs to be the same across source buffers. It means that establishing such offsets needs to be done explicitly by the application or, if they are derived from timestamps in the media it needs to be done in a consistent way (in terms of which out of audio and video the time offset is taken from).

I think this has implications for the push/pop time offset as well. They should be global methods which establish a global offset based on the next-appended segment(s).

We do also need a way to handle the user starting content in the middle. If I have a 30 min content item and the user wants to start at minute 10 (because of a bookmark, say) then I should be able to start appending data at position 10min in the source buffer timeline. The seek bar needs to show the playback starting at minute 10 and if the user seeks backwards this should be ok.

pushOffset isn't right for this case because the media internal timestamps are correct: the first segment appended really does start at timestamp 10min.

I wonder whether we should just say that the source buffer timestamp starts at zero and not derive a start point from the appended media. If the media internal timestamp corresponding to the start of the content is not zero you need to explicitly handle this with a pushOffset call ?


Applications could also just append the segments to a "scratch" SourceBuffer to see what the initial timestamp is and then use that information to compute the proper offset to apply. It's not the greatest solution, but it does provide a way for people to handle this if they aren't as careful about how they create their demuxed content.


Aaron


Hmm - no clear answer here - I'll think about this some more.

...Mark


Aaron