[media-source] Clarify that overlapped video coded frames need their duration adjusted in the track buffer

wolenetz has just created a new issue for 
https://github.com/w3c/media-source:

== Clarify that overlapped video coded frames need their duration 
adjusted in the track buffer ==
Imaging this scenario:
SourceBuffer.appendBuffer({video initialization segment});
SourceBuffer.appendBuffer({video coded frame keyframe DTS=PTS=0, 
duration = 30ms});
SourceBuffer.appendBuffer({video coded frame keyframe DTS=PTS=20, 
duration = 30ms});

video track buffer per v1 spec would contain: frames [0,30)[20,50). 
Note the overlap.

A subsequent SourceBuffer.remove(20,30) would remove the coded frame 
[20,50), resulting in track buffer containing: [0,30).

This may be confusing to web authors and implementors, in that an 
overlapped video frame, the duration isn't adjusted in the track 
buffer, and rendering that frame might or might not observe the 
duration. Further, if it *doesn't* observe that duration, rather it 
respects the overlapping video frame's PTS, then it could be confusing
 when a removal of the overlapping frame makes the originally 
overlapped frame's presentation interval "spring back" to 30ms in this
 example.

I suggest that we clarify in the spec the behavior really should be to
 adjust the duration of the overlapped frame (similar to that in audio
 splice frame). Text might be:

Current:
* If track buffer contains video coded frames:
 * Let remove window timestamp equal the overlapped frame presentation
 timestamp plus 1 microsecond.
 * If the presentation timestamp is less than the remove window 
timestamp, then remove overlapped frame from track buffer.

Suggested new text:

* If track buffer contains video coded frames:
 * Let remove window timestamp equal the overlapped frame presentation
 timestamp plus 1 microsecond.
 * If the presentation timestamp is less than the remove window 
timestamp,
  * then remove overlapped frame from track buffer.
  * otherwise, update the overlapped frame in the track buffer's to 
have coded frame duration set to difference between presentation 
timestamp and the overlapped frame presentation timestamp.

Chrome currently does the "otherwise" case for certain byte streams 
(webm) when those byte streams don't explicitly set a precise coded 
frame duration (e.g., a SimpleBlock frame at the end of a cluster), so
 the duration is initially estimated. Such estimated durations are 
"fixed up" on overlapped appends to them, but that leaves the 
non-estimated-duration webm blocks, as well as any non-webm coded 
frames to suffer from having overlapping presentation intervals in the
 same track buffer in the scenario, above.

This clarification might best be done at this point in VNext, given 
the V1 spec is nearing PR/REC.

Please view or discuss this issue at 
https://github.com/w3c/media-source/issues/166 using your GitHub 
account

Received on Thursday, 22 September 2016 22:43:12 UTC