Re: TextTrack questions

Comments inline...


On Tue, Jan 21, 2014 at 7:06 PM, Philip Jägenstedt <philipj@opera.com>wrote:

> On Wed, Jan 22, 2014 at 3:31 AM, Aaron Colwell <acolwell@google.com>
> wrote:
> >
> > On Tue, Jan 21, 2014 at 8:05 AM, Eric Carlson <eric.carlson@apple.com>
> > wrote:
> >>
> >>
> >> On Jan 17, 2014, at 6:38 PM, Philip Jägenstedt <philipj@opera.com>
> wrote:
> >>
> >> > I hadn't thought about those case either. It would be nice (simple) to
> >> > just say that for all in-band tracks, when the buffered ranges are
> >> > updated, evict all cues which now do not overlap the buffered ranges.
> >> > Is that going to break something?
> >> >
> >>
> >>   That does seem like the simplest (to implement and understand))
> solution
> >> to this issue.
> >>
> >>   I don’t know of anything that it will break.
> >
> >
> > I have a few questions about this:
> >
> > 1. Does "evict all cues" include ones added by JavaScript or just the
> ones
> > added by the inband track?
>
> All cues. If you don't want to lose them, put them in a script-created
> track or keep them around to re-insert later when that regions becomes
> buffered again.
>

Agreed.


>
> > 2. For out-of-band tracks is the application expected to remove cues from
> > unbuffered regions? I'm mainly thinking about a live broadcast where cues
> > are inserted by JavaScript instead of in-band.
>
> I don't think we should throw away cues for out-of-band or
> script-created tracks, no. The application will have to throw them
> away if it's an infinite stream.
>

Makes sense to me. I just wanted to double check.


>
> > 3. I'm assuming "when the buffered ranges are updated" is evaluated on a
> per
> > track basis and not actually what the HTMLMediaElement exposes since
> removal
> > of text track data does not necessarily imply removal of audio/video
> data.
> > Am I understanding the intended meaning correctly?
>
> I did mean HTMLMediaElement.buffered. AudioTrack/VideoTrack/TextTrack
> don't expose buffered ranges, so I'm not sure what the alternative
> would be. Do you mean something with SourceBuffer.buffered or some
> spec-internal concept I'm forgetting?
>

The changes I'm talking about would not necessarily be script visible
because they are replacing existing buffered data in the presentation. This
would likely be done atomically from JavaScript's perspective so script
would have no opportunity to observe a hole in buffered. Also because text
tracks are discontinuous they don't effect buffered ranges in the same way
that audio or video would. For example if I have a time region that
contains audio & a few cues and I overlap that with a media segment that
only contains audio, the cues should be removed, but no hole in the
buffered ranges would appear because the new data completely replaces the
old data.

Because text tracks are discontinuous, they can't really effect the
buffered ranges in the way that audio & video do. If we did a simple
intersection of all buffered ranges across tracks the result would only
show buffered data where cues are. That is clearly wrong. In Chromium's MSE
implementation we treat inband text track buffered ranges as a single range
between [0-duration) so that simple intersection with the audio & video
tracks buffered ranges will show the actual buffered ranges that are
playable. Given this, any changes to an in-band text track won't effect the
buffered ranges eventhough the actual cues in the presentation have changed.


>
> > 4. MSE overlapping appends should be treated as a time range becoming
> > "unbuffered", so old cues are removed, and then "buffered" with the new
> cues
> > being appended. Right?
>
> So, assuming that an overlapping append causes
> HTMLMediaElement.buffered to modified at a time when scripts can
> observe it, yes, there would be an opportunity to evict cues.
> Basically, I think the HTML spec should have a hook that is run
> anytime that a script accessing .buffered would see something
> different than it did before that point. I haven't checked if that's
> implementable in an efficient way though, if one calculates the
> buffered ranges lazily on request then we have a problem...
>

As I've outlined above, I don't think we can simply restrict this to
observable changes in buffered. I do think it would be handy to have a hook
in the HTML spec that can be invoked when a buffered time range has been
removed from the presentation. That would at least allow there to be a step
in the MSE spec which triggers the removal of old cues when an overlapping
append occurs. There could also be language in there about the buffered
attribute being updated, but the MSE spec overloads the default buffered
behavior so it wouldn't apply to MSE.

I hope this helps. Thanks for helping me figure this stuff out. :)

Aaron


>
> Philip
>