Re: Dropping cues when playbackRate != 1.0

On Aug 21, 2013, at 5:12 PM, Silvia Pfeiffer <silviapfeiffer1@gmail.com> wrote:

> 
> 
> 
> On Sun, Aug 18, 2013 at 12:42 AM, Eric Carlson <eric.carlson@apple.com> wrote:
> 
> 
> On Aug 17, 2013, at 5:40 AM, Silvia Pfeiffer <silviapfeiffer1@gmail.com> wrote:
> 
>> 
>> 
>> 
>> On Sat, Aug 17, 2013 at 3:17 AM, Eric Carlson <eric.carlson@apple.com> wrote:
>> 
>> On Aug 8, 2013, at 7:21 PM, Silvia Pfeiffer <silviapfeiffer1@gmail.com> wrote:
>> 
>> > On Fri, Aug 9, 2013 at 6:11 AM, Brendan Long <self@brendanlong.com> wrote:
>> >> There's section of the HTML5 CR spec saying:
>> >>
>> >> Similarly, when the playback rate is not exactly 1.0, hardware, software, or
>> >> format limitations can cause video frames to be dropped and audio to be
>> >> choppy or muted.
>> >>
>> >>
>> >> I think this should also apply to text track cues, something like:
>> >>
>> >> Similarly, when the playback rate is not exactly 1.0, hardware, software, or
>> >> format limitations can cause video frames and text cues to be dropped and
>> >> audio to be choppy or muted.
>> >>
>> >> When playing at non-standard speeds, an efficient implementation may want to
>> >> skip portions of the file, which could mean skipping cues.
>> >
>> > I assume you are talking about a situation where the playback would
>> > need to jump multiple seconds (call them "s") at a time to achieve the
>> > speedup.
>> > And you are further assuming that when the current time jumps from
>> > second x to second x+s, there may be a cue that would have both their
>> > start and end time between x and x+s.
>> > So you're saying that if there is no audio or video rendered to which
>> > the cue refers, the "time marches on" algorithm doesn't actually
>> > activate these cues and therefore they are "dropped".
>> >
>> > In actual fact, the "time marches on" algorithm will activate the
>> > enter and exit events of the "missed cues", so they are not really
>> > "dropped". They are, however, not rendered, because they don't become
>> > active.
>> >
>>   This requires that all of the media data for time X to X+S has to be loaded before skipping ahead.
>> 
>> Why? Only the media data for time x and for time x+s has to be loaded - nothing else.
>> 
>>  
>> This means that a UA can not play at a faster rate by not loading the entire file, for example by only loading and displaying key frames. 
>> 
>>   If this is allowed, and it seems silly to prevent it, cues defined in the portions of the file that are not loaded would have to be skipped.
>> 
>> I guess we have two different kind of cues: those coming from <track> and those coming from inside a media file.
>> 
>> Those from a <track> element are all loaded and events could be activated as the timeline skips over the cue start/end times. Cue content would be rendered when X or X+S is within a cue's interval.
>> 
>> Those from inside a media file, if multiplexed, may get completely skipped by going from X to X+S without the browser loading those parts of a file. So, in this case you'd probably skip cues.
>> 
> 
>   Yes, sorry for being unclear, I was talking about in-band captions. 
> 
> 
> I'm trying to figure out whether this may be a problem, both because it's inconsistent between in-band and <track> based text tracks, and because JS developers may need to be notified of skipped cues.
> 
> In your experience, for in-band tracks, would it be possible to require that the browser not fast-forward text tracks, but only skip audio / video track sections?
> 
  Not unless we disallow seeking to an un-buffered time. There is no chance I would make a change like that.

> I guess it would require writing some sort of index of the cues into the file. It's possible in Ogg with a special Skeleton header for text tracks.
> 
  That won't help because the index will only give you the cue timestamps, not the text.

eric