W3C home > Mailing lists > Public > public-html-media@w3.org > July 2012

Re: [MSE] Timestamp offset mechanism

From: Mark Watson <watsonm@netflix.com>
Date: Tue, 24 Jul 2012 20:09:53 +0000
To: Aaron Colwell <acolwell@google.com>
CC: "public-html-media@w3.org" <public-html-media@w3.org>
Message-ID: <8A9DBC72-7BA2-4759-A549-77E0E65CAA00@netflix.com>
Sounds good to me too.

I think there are multiple issues whenever the UA tries to derive something 'automatically' from the media timestamps. This applies at startup too, which is the subject of another thread.

It may be simpler, in the first instance, to require the application always to know what the media timestamps are, and to set timestampOffset appropriately.

…Mark

On Jul 24, 2012, at 11:08 AM, Aaron Colwell wrote:

Ping. I haven't seen any responses to my last email on this thread so I'm not sure how to proceed.

I'm retracting my support for the push/popTimestampOffset() proposal until someone can address concerns I mentioned.

For now I'm proposing we just do the simplest thing that would work:
1. Add  an  "attribute double timestampOffset" to SourceBuffer.
2. The value of this attribute is added to media timestamps before the frames are inserted into the source buffer. 0 is the default value for this attribute since that results in unmodified timestamps.
3. If the app tries to set this attribute while in the middle of appending a media segment, an INVALID_STATE_ERR exception is thrown and the attribute is not updated. If the app doesn't know whether it is in the middle of a media segment, it can always call abort() to reset the parser state.

Please let me know if you are ok with this simple attribute proposal or whether you have arguments for the push/pop mechanism that I should reconsider.

Thanks,
Aaron


On Wed, Jul 18, 2012 at 2:19 PM, Aaron Colwell <acolwell@google.com<mailto:acolwell@google.com>> wrote:
So I'm starting to waver on my support for the pushTimestampOffset() & popTimestampOffset() mechanism and leaning towards the simple timestampOffset attribute again.

Here is why:
1. The offset stack can be implemented in JavaScript. I'm not really seeing an advantage for putting this in the UA.
2. Mark's example looks like a common use case, but it only pushes things onto the stack. This is essentially equivalent to updating a timestampOffset attribute.
3. If the application isn't aware of the actual internal timestamps of a media segment, it could always append it to a "scratch" SourceBuffer on a different MediaSource object and discover the start timestamp by calling SourceBuffer.buffered.

Am I missing something key here? Can someone provide other examples that show why it is advantageous for the UA to maintain the stack?

Aaron


On Thu, Jul 12, 2012 at 4:45 PM, Aaron Colwell <acolwell@google.com<mailto:acolwell@google.com>> wrote:
Ok. This seems reasonable to me.

Question 2: I was just trying to determine if we wanted to provide a way for the app to "rediscover" that an offset is currently active in case it wasn't keeping track. Perhaps:

// Number levels of timestampOffset (ie how many popTimestampOffset() I need to clear them all)
readonly attribute int timestampOffsetDepth;

or

// Indicates at least one timestamp offset is on the offset stack.
readonly attribute bool usingTimestampOffset;

I'm not sure how critical this is, but I could definitely see people getting confused if they forgot that they called pushTimestampOffset() and then had no way to detect that an offset is still being applied.


Question 4: Yes I think that an offset should stay in place across multiple segments. I was thinking about events that would fire when push & pop actions actually become active. For example if you pushed an offset while in the middle of appending a media segment, it wouldn't take effect until the end of that segment. Once the segment ended a 'timestampoffsetstart' event would fire since that is when the new offset would start being applied. If a pop happens mid-segment then a 'timestampoffsetend' would fire when the current segment has been completely appended.

I'm not sure whether these events are really that useful or not, but I just wanted to ask the question to see what people thought.

Aaron

On Thu, Jul 12, 2012 at 2:33 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:

On Jul 12, 2012, at 1:59 PM, Mark Watson wrote:


On Jul 12, 2012, at 12:58 PM, Aaron Colwell wrote:

Yes. The idea was to prevent the app from having to worry about the internal timestamp and only worry about the presentation times. I think this API needs some polishing though.

Here are some outstanding questions I have:

1. nextSegmentStartTime() & clearTimestampOffset() names are a little clunky and don't seem related. Alternate name ideas anyone?

pushTimestampOffset() and popTimestampOffset() ?


2. How does the app determine that an offset is currently being applied?

Does it need too ? Since the app is the only one which can establish one.

peekTimestampOffset() ?


3. Should we model this as offset stack? I think Adrian suggested this during the last Media Source telecom.

Yes.


4. Should events be defined to signal when an offset starts/stops being applied? Effectively this marks the beginning of the next media segment after either method is called.'

Hmm. I assumed the offset persists across multiple media segments until you cancel it.

You may, for example, have 200 segments of main media content that start at timestamps 0, 2, 3, … 398s. You feed in 100 of them without setting an offset. Then you have 15 ad segments with timestamps 0, … 28s and before feeding these in you call pushTimestampOffset( 200s ). Once finished with these you have to call popTimestampOffset() and pushTimestampOffset( 30 ) before feeding in the rest of the media segments from the main content.

Oops, I mean pushTimestampOffset( 230s ) for the last one here as we are specifying the actual source buffer time where we want the next segment to appear, not the offset between internal media time and source buffer time.



5. Does anyone have an alternate proposal for applying offsets?

Aaron

On Wed, Jul 11, 2012 at 9:26 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
Actually, I liked the nextSegmentStartTime (option 3). That is, assuming I remember correctly that what it does: establish a mapping from internal media time to source buffer time such that the next segment is placed at the provided source buffer time. (strictly, the mapping is established when the next segment arrives).

This obviates the need for the app to understand much about the internal media time.

...Mark

Sent from my iPhone

On Jul 10, 2012, at 11:50 AM, "Aaron Colwell" <acolwell@google.com<mailto:acolwell@google.com>> wrote:

Hi All,

I'm going to try to summarize what I've read on this thread to see if I understand everyone's position

1. Eric, Steve, & Kevin all support the sourceTimestampOffset() signature
2. Mark doesn't have a strong preference either way, but is leaning towards sourceTimestampMapping() because it is more explicit about the application needing to know about the internal timestamps in the media segments and how they should map to the presentation timeline.
3. The alternative nextSegmentStartTime()/clearTimestampOffset() suggestion mentioned at the bottom of http://lists.w3.org/Archives/Public/public-html-media/2012Jun/0095.html got no votes
4. Duncan feels this mechanism isn't sufficient to address ad-insertion use cases he cares about. I just started an Ad Insertion thread<http://lists.w3.org/Archives/Public/public-html-media/2012Jul/0033.html> to continue the discussion. This may require adding a new feature.

Let me know if this is correct or if I've missed something.

Thanks,
Aaron


On Fri, Jun 15, 2012 at 3:40 PM, Aaron Colwell <acolwell@google.com<mailto:acolwell@google.com>> wrote:
Hi,

Bug 17004<https://www.w3.org/Bugs/Public/show_bug.cgi?id=17004> proposes a mechanism for adjusting timestamps inside media segments. This enables seamlessly splicing content together that doesn't originally exist on a common presentation timeline. Inserting advertisements into content is the most straight forward use case. There has been some useful discussion in the bug comments, but we haven't decided on which method signature to use for specifying the timestamp adjustments.

Here are the proposed signatures at this point:

void sourceTimestampMapping(in double presentationTimestamp, in double segmentTimestamp)
 -- or --
void sourceTimestampOffset(in double timestampOffset)

sourceTimestampMapping() just provides a way to say, "I want this timestamp in the segment to map to this presentation time."
sourceTimestampOffset() is basically just saying, "Add this offset to any timestamp in the segment to get the desired presentation time."

Fundamentally they do the exact same thing, but I'm looking for input on which option people feel more comfortable with.

Which do you prefer?
Are you ok with the semantics outlined in the bug description<https://www.w3.org/Bugs/Public/show_bug.cgi?id=17004#c0>?

Aaron
Received on Tuesday, 24 July 2012 20:10:23 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 15:48:24 UTC