- From: <bugzilla@jessica.w3.org>
- Date: Mon, 25 Feb 2013 22:39:06 +0000
- To: public-html-media@w3.org
https://www.w3.org/Bugs/Public/show_bug.cgi?id=21112
Bug ID: 21112
Summary: buffer/splice/overlap model constrains natural editing
granularity to whole media segments
Classification: Unclassified
Product: HTML WG
Version: unspecified
Hardware: PC
OS: All
Status: NEW
Severity: normal
Priority: P2
Component: Media Source Extensions
Assignee: adrianba@microsoft.com
Reporter: mthornbu@adobe.com
QA Contact: public-html-bugzilla@w3.org
CC: mike@w3.org, public-html-media@w3.org
the current Media Source Extensions specification's buffering/splicing/overlap
model for appending media segments has a new append start at the beginning of
the media segment and continue to its end (potentially overlaying a portion of
media already in the buffer).
consider programs A (the "main program") and B (the "ad"), with A being live.
the stream encoder/segmenter will typically be free-running, making random
access points and segment boundaries in natural places independent of any
external cue inputs. an operator may at some point push the "ad goes here"
button, which should only have to create a cue marker in the manifest file. it
may be impractical or infeasible to affect the operation of the
encoder/segmenter to create a segment boundary at the ad-start or
ad-end-and-main-program-resumes points.
0s 14s 31s 42s
+-- cue B +-- cue A
prog A v v
|-----------|----:vvvvvv|. . . . .|vvvvvvvvvv:---|-----------|-----------|
A1(1) A2 : A3(-) A4(4) : A5(7) A6(8)
(2) :B1(3) B2(5) B3(6) :
|---------|---------|-------|
prog B
0s 28s
1. append A1;
2. append A2;
3. append B1 at +14s in;
4. append A4;
5. append B2 at +14s in;
6. append B3 at +14s in;
7. append A5;
8. append A6...
in this example, main program segment A4 is overlapped by ad segments B2 and
B3. this can be accommodated with the current buffering/overlap model, but in
a fairly unnatural way. to achieve the desired rendering, the append order
must be [A1, A2, B1, A4, B2, B3, A5, A6, ...] -- in other words, not in the
natural playback order. every application will need to implement a segment
overlap scheduler to get this ordering right. note also that there is a race
with the playback position vs the appends, where if you're running close to the
playback position, you might display a portion of the wrong program (for
example, missing the beginning of an ad or temporarily switching back to the
main program in the middle of the ad).
this works for the ad insertion case because the advertiser will typically want
their entire ad played from beginning to end. for the general "video editing"
case, there's no way to come in to program B at not-a-segment-boundary from
program A not-a-segment-boundary, using the current model.
it would be more natural to be able to append the media segments in the order
they are to be rendered, but to specify slices or windows of the media segments
to be appended to the timeline. in other words:
0s 14s 0s 11s
+-- cue B +-- cue A
prog A v v
|-----------|----:XXXXXX|. . . . .|>>>>>>>>>>:---|-----------|-----------|
A1(1) A2 : A3(-) A4(6) : A5(7) A6(8)
(2) :B1(3) B2(4) B3(5) :
|---------|---------|-------|
prog B
0s 28s
1. append A1;
2. append A2 until 14s;
3. append B1;
4. append B2;
5. append B3;
6. append A4 from 11s in;
7. append A5;
8. append A6...
so append order would be [A1, portion of A2, B1, B2, B3, portion of A4, A5, A6,
...].
Aaron Colwell proposed on the mailing list adding an "append window" API:
partial interface SourceBuffer {
attribute double appendWindowStart;
attribute unrestricted double appendWindowEnd;
}
- appendWindowStart is initially set to 0;
- appendWindowEnd is initially set to positive Infinity.
- Setting appendWindowStart throws an exception if one tries to set it to a
value >= appendWindowEnd.
- Setting appendWindowEnd throws and exception if one tries to set it to a
value <= appendWindowStart.
- The attributes can only be modified when updating == false, just like
timestampOffset.
- abort() resets appendWindowStart to 0 and appendWindowEnd to +Infinity
with additional proposed filtering behavior (should be discussed more):
- The coded frame processing algorithm drops coded frames w/
presentationTimestamp < appendWindowStart
- The coded frame processing algorithm drops coded frames w/
presentationTimestamp >= appendWindowEnd.
- If a coded frame is dropped before appendWindowStart, then a "needs RAP" flag
is set so that the coded frame processing algorithm will continue to drop coded
frames until it receives a RAP with a presentation timestamp >=
appendWindowStart.
the above additional proposed filtering algorithm may not yield the same
results as is possible with out-of-natural-order overlapped appends. i propose
wording the algorithm in terms of windowing rather than coded frame filtering
such that an implementor could choose a more complex implementation that would
not require random access point boundaries, but would allow an implementation
to do that and leave a gap in the display as is currently the case with the
overlap model.
i additionally propose that these properties should interact favorably with
whatever solution is found for bug 20901 "contiguous splice/append without
knowing media segment internal timestamps" such that a discontinuous append
shall commence into the timeline from the appendWindowStart.
--
You are receiving this mail because:
You are on the CC list for the bug.
Received on Monday, 25 February 2013 22:39:08 UTC