RE: [MSE] New Proposal for Bug 20901

hi Aaron.

for Use Case D, if the transport stream "discontinuity indicator" is what's used for detecting "segment boundaries", then that could work. however, since transport streams don't have natural and detectable "segment boundaries", a manual signal would still be needed for TS segments when their internal timestamps are discontinuous from a first segment to a second but the TS files don't include a discontinuity indication.  unless a heuristic is developed for the TS-specific section to detect a discontinuity by observing the timestamps, and seeing that they either go backwards by more than X or jump forward more than Y vs the observed interframe period.

otherwise this seems like it could work and is no more complicated than the previous proposal for when there are multiple SourceBuffers that need to be synchronized: one would be chosen to be the master and set to "sequence", abort() called (for TS segments), the append done for the master, the new timestampOffset recovered from that. then that timestampOffset can be used on the other SourceBuffers in "segments" mode (after an abort() for TS segments) to have those buffers synchronized with the master by timestamps.

i think having the AppendMode's default value vary based on the media type, considering both modes will probably needed in a real application, is surprising (for the JS programmer) and fragile.  i think the AppendMode's initial value should be the same in all cases.

-michael thornburgh


From: Aaron Colwell [mailto:acolwell@google.com]
Sent: Wednesday, May 01, 2013 3:40 PM
To: <public-html-media@w3.org>
Subject: [MSE] New Proposal for Bug 20901

At the F2F meeting last week I agreed to craft a new proposal for handling out-of-order appends, appends w/o internal timestamp knowledge, and discontinuities in MPEG2 TS. I'm going to start out by outlining the relevant use cases as I understand them and then propose the new solution that I believe will address all these use cases and avoid the issues with the current solution.

Use Cases:
Use Case A: Out-of-order media segment appends.
 Formats like ISOBMFF and WebM have self-contained media segments with clear start and end boundaries. Applications should be able to append these segments in any order. These segments can be split across many appendXXX() calls without any ambiguity because all coded frames within a media segment are guaranteed to be adjacent in the presentation.

Use Case B: Appending media into a continuous sequence w/o knowledge of internal timestamps.
 Some applications want to create a presentation by concatenating media segments from different sources without knowledge of the timestamps inside the segments. Each media segment appended should be placed, in the presentation timeline, immediately after the previously appended segment independent of what the internal timestamps are. At the beginning of each media segment, a new timestampOffset value is calculated so that the timestamps in the media segment will get mapped to timestamps that immediately follow the end of the previous media segment.

Use Case C: Place media at a specific location in the timeline w/o knowledge of internal timestamps.
 This is related to Use Case B. This case is useful for placing media segments from a third party in the middle of a presentation. It also allows an application that receives media segments from a live source to easily map the first segment received to presentation time 0.

Use Case D: Handling of MPEG2-TS discontinuities without application intervention.
 MPEG2-TS streams are allowed to have timestamp discontinuities that can make it difficult for the UA to detect out-of-order appends when the discontinuity is split across two appendXXX() calls. The web application likely doesn't know where in the bytestream these discontinuities occur so MSE needs to provide a mode where a sequence of appendXXX() calls are always considered to be adjacent. In this mode the application needs to provide an explicit signal, like an abort() call, to indicate that the next appendXXX() is not adjacent to the previous calls.



How the current spec text addresses the use cases:
Use Case A :  abort() must be called before an out-of-order append. Step 7 of the coded frame processing algorithm<https://dvcs.w3.org/hg/html-media/raw-file/ffb76048861e/media-source/media-source.html#sourcebuffer-coded-frame-processing> triggers an error if an out-of-order append occurs without an abort().

Use Case B : abort("continuation") must be called between media segments that don't have internal timestamps that are adjacent.

Use Case C : abort("timestampOffset") must be called and timestampOffset is updated to the desired starting timestamp. The next appendXXX() call will trigger timestampOffset to get updated so that the media segment will get inserted at the desired location.

Use Case D : A sequence of appendXXX() calls w/o any abort() call between them are considered adjacent. This is what the spec calls an 'append sequence'.


New Proposal:
Here is a new proposal that doesn't make supporting the use cases rely on abort() calls:

- Remove AbortMode parameter from abort().

- Add the following AppendMode enum
  enum AppendMode {
      "segments"
      "sequence",
  };
  * The "segments" mode indicates that media segments can appended in any order and no assumption about adjacency is made.
  * The "sequence" mode indicates that media segments are assumed to be adjacent in presentation time irrespective of the internal timestamps. A new media segment is appended immediately after the previously appended segment and the timestampOffset attribute is updated accordingly to reflect the offset that is being applied to the internal timestamps.

- Add an mode attribute with the following signature.
  partial interface SourceBuffer {
    AppendMode mode;
  }
  * mode is set  to a "default append mode" when the SourceBuffer is created. This could be defined by the bytestream format. WebM & ISO-BMFF would specify "segments" as the default and MPEG2-TS would specify "sequence".
  * mode can only be updated while not updating (ie updating == false).

- If timestampOffset is set while mode == "sequence" then the next appendXXX() call will place the content at the specified timestamp offset. timestampOffset will get updated by this append to reflect what offset that is being applied to the internal timestamps to place the media segment at the desired location.


How the new proposal addresses the use cases:

Use Case A: The "segments" mode allows out-of-order appends w/o abort() calls.

Use Case B: The "sequence" mode ensures that segments are appended one after the other independent of their internal timestamp. timestampOffset is updated to the current offset being applied to the internal timestamps so that the application can properly synchronize appends across SourceBuffers.

Use Case C: The "sequence" mode plus updated timestampOffset setter behavior addresses this use case.

Use Case D: MPEG2-TS streams will automatically be placed in the "sequence" mode which allows discontinuities to be properly handled because this mode is equivalent to the "append sequence" in the current spec.


I think this new proposal make things simpler for web applications and drops the requirement of abort() calls for Use Case A. If someone objects to the proposal or has questions please let me know as soon as possible. I'd like to get these changes applied before the next Media TF call.

Aaron