Re: [MSE] New Proposal for Bug 20901

Hi Michael,

Thanks for your response. Comments inline...


On Wed, May 1, 2013 at 4:17 PM, Michael Thornburgh <mthornbu@adobe.com>wrote:

> hi Aaron.****
>
> ** **
>
> for Use Case D, if the transport stream “discontinuity indicator” is
> what’s used for detecting “segment boundaries”, then that could work.
> however, since transport streams don’t have natural and detectable “segment
> boundaries”, a manual signal would still be needed for TS segments when
> their internal timestamps are discontinuous from a first segment to a
> second but the TS files don’t include a discontinuity indication.  unless a
> heuristic is developed for the TS-specific section to detect a
> discontinuity by observing the timestamps, and seeing that they either go
> backwards by more than X or jump forward more than Y vs the observed
> interframe period.
>

I think a heuristic will likely be used that is similar to Step 7 of the
coded frame processing algorithm in the current spec. Instead of triggering
an error condition, it would signal that a discontinuity was detected and
act accordingly. I think this should work, but I haven't sorted out all the
details yet. I plan to start working on the spec updates today so I should
have a better sense whether this is a viable solution by EOD.

My current thinking is that unmarked discontinuities in MPEG2-TS are
essentially the same as out-of-order appends in the "sequence" mode. If the
UA notices that the timestamps in the coded frames aren't immediately after
the previous ones, then a discontinuity has occurred and the
timestampOffset needs to be updated accordingly so that the new coded
frames get placed right after the previous ones. I believe there are a few
edge cases I need to watch out for in muxed content, but I'll have a better
handle on that once I've started updating the spec.


> ****
>
> ** **
>
> otherwise this seems like it could work and is no more complicated than
> the previous proposal for when there are multiple SourceBuffers that need
> to be synchronized: one would be chosen to be the master and set to
> “sequence”, abort() called (for TS segments), the append done for the
> master, the new timestampOffset recovered from that. then that
> timestampOffset can be used on the other SourceBuffers in “segments” mode
> (after an abort() for TS segments) to have those buffers synchronized with
> the master by timestamps.****
>
> **
>

Yes. I agree.


> **
>
> i think having the AppendMode’s default value vary based on the media
> type, considering both modes will probably needed in a real application, is
> surprising (for the JS programmer) and fragile.  i think the AppendMode’s
> initial value should be the same in all cases.****
>
> **
>

I was on the fence about this. I figured I'd throw it in there to see what
people thought. I'm fine with having a fixed initial value. For now I'll
just specify the initial mode to "segments" since that is what 2 out of the
3 bytestream formats support. I expect that good arguments could be made
for making "sequence" the initial mode, but I'll defer that until the spec
has been updated with the new proposal.

Thanks again for your feedback. :)

Aaron



> **
>
> -michael thornburgh****
>
> ** **
>
> ** **
>
> *From:* Aaron Colwell [mailto:acolwell@google.com]
> *Sent:* Wednesday, May 01, 2013 3:40 PM
> *To:* <public-html-media@w3.org>
> *Subject:* [MSE] New Proposal for Bug 20901****
>
> ** **
>
> At the F2F meeting last week I agreed to craft a new proposal for
> handling out-of-order appends, appends w/o internal timestamp knowledge,
> and discontinuities in MPEG2 TS. I'm going to start out by outlining the
> relevant use cases as I understand them and then propose the new solution
> that I believe will address all these use cases and avoid the issues with
> the current solution.****
>
> ** **
>
> *Use Cases:*****
>
> *Use Case A: Out-of-order media segment appends.*****
>
>  Formats like ISOBMFF and WebM have self-contained media segments with
> clear start and end boundaries. Applications should be able to append these
> segments in any order. These segments can be split across many appendXXX()
> calls without any ambiguity because all coded frames within a media segment
> are guaranteed to be adjacent in the presentation.****
>
> ** **
>
> *Use Case B: Appending media into a continuous sequence w/o knowledge of
> internal timestamps.*****
>
>  Some applications want to create a presentation by concatenating
> media segments from different sources without knowledge of the timestamps
> inside the segments. Each media segment appended should be placed, in the
> presentation timeline, immediately after the previously appended segment
> independent of what the internal timestamps are. At the beginning of each
> media segment, a new timestampOffset value is calculated so that the
> timestamps in the media segment will get mapped to timestamps that
> immediately follow the end of the previous media segment.****
>
> ** **
>
> *Use Case C: Place media at a specific location in the timeline w/o
> knowledge of internal timestamps.*****
>
>  This is related to Use Case B. This case is useful for placing media
> segments from a third party in the middle of a presentation. It also allows
> an application that receives media segments from a live source to easily
> map the first segment received to presentation time 0.****
>
> ** **
>
> *Use Case D: Handling of MPEG2-TS discontinuities without
> application intervention.*****
>
>  MPEG2-TS streams are allowed to have timestamp discontinuities that can
> make it difficult for the UA to detect out-of-order appends when the
> discontinuity is split across two appendXXX() calls. The web application
> likely doesn't know where in the bytestream these discontinuities occur so
> MSE needs to provide a mode where a sequence of appendXXX() calls are
> always considered to be adjacent. In this mode the application needs to
> provide an explicit signal, like an abort() call, to indicate that the next
> appendXXX() is not adjacent to the previous calls.****
>
> ** **
>
> ** **
>
> ** **
>
> *How the current spec text addresses the use cases:*****
>
> Use Case A :  abort() must be called before an out-of-order append. Step 7
> of the coded frame processing algorithm<https://dvcs.w3.org/hg/html-media/raw-file/ffb76048861e/media-source/media-source.html#sourcebuffer-coded-frame-processing> triggers
> an error if an out-of-order append occurs without an abort().****
>
> ** **
>
> Use Case B : abort("continuation") must be called between media segments
> that don't have internal timestamps that are adjacent.****
>
> ** **
>
> Use Case C : abort("timestampOffset") must be called and timestampOffset
> is updated to the desired starting timestamp. The next appendXXX() call
> will trigger timestampOffset to get updated so that the media segment will
> get inserted at the desired location.****
>
> ** **
>
> Use Case D : A sequence of appendXXX() calls w/o any abort() call between
> them are considered adjacent. This is what the spec calls an
> 'append sequence'.****
>
> ** **
>
> ** **
>
> *New Proposal:*****
>
> Here is a new proposal that doesn't make supporting the use cases rely
> on abort() calls:****
>
> ** **
>
> - Remove AbortMode parameter from abort().****
>
> ** **
>
> - Add the following AppendMode enum****
>
>   enum AppendMode {****
>
>       "segments"****
>
>       "sequence",****
>
>   };****
>
>   * The "segments" mode indicates that media segments can appended in any
> order and no assumption about adjacency is made.****
>
>   * The "sequence" mode indicates that media segments are assumed to be
> adjacent in presentation time irrespective of the internal timestamps. A
> new media segment is appended immediately after the previously appended
> segment and the timestampOffset attribute is updated accordingly to reflect
> the offset that is being applied to the internal timestamps.****
>
> ** **
>
> - Add an mode attribute with the following signature.****
>
>   partial interface SourceBuffer {****
>
>     AppendMode mode;****
>
>   }****
>
>   * mode is set  to a "default append mode" when the SourceBuffer is
> created. This could be defined by the bytestream format. WebM & ISO-BMFF
> would specify "segments" as the default and MPEG2-TS would specify
> "sequence".****
>
>   * mode can only be updated while not updating (ie updating == false).***
> *
>
> ** **
>
> - If timestampOffset is set while mode == "sequence" then the next
> appendXXX() call will place the content at the specified timestamp
> offset. timestampOffset will get updated by this append to reflect what
> offset that is being applied to the internal timestamps to place the media
> segment at the desired location.****
>
> ** **
>
> ** **
>
> *How the new proposal addresses the use cases:*****
>
> ** **
>
> Use Case A: The "segments" mode allows out-of-order appends w/o abort()
> calls.****
>
> ** **
>
> Use Case B: The "sequence" mode ensures that segments are appended one
> after the other independent of their internal timestamp. timestampOffset
> is updated to the current offset being applied to the internal timestamps
> so that the application can properly synchronize appends across
> SourceBuffers.****
>
> ** **
>
> Use Case C: The "sequence" mode plus updated timestampOffset setter
> behavior addresses this use case.****
>
> ** **
>
> Use Case D: MPEG2-TS streams will automatically be placed in the
> "sequence" mode which allows discontinuities to be properly handled because
> this mode is equivalent to the "append sequence" in the current spec.****
>
> ** **
>
> ** **
>
> I think this new proposal make things simpler for web applications and
> drops the requirement of abort() calls for Use Case A. If someone objects
> to the proposal or has questions please let me know as soon as possible.
> I'd like to get these changes applied before the next Media TF call.****
>
> ** **
>
> Aaron****
>
> ** **
>