Re: TextTrack questions

On 17 Jan 2014 06:00, "Aaron Colwell" <acolwell@google.com> wrote:
>
> Comments inline...
>
> On Wed, Jan 15, 2014 at 2:44 PM, Silvia Pfeiffer <
silviapfeiffer1@gmail.com> wrote:
>>
>> Hi Aaron,
>>
>> On Thu, Jan 16, 2014 at 9:02 AM, Aaron Colwell <acolwell@google.com>
wrote:
>> > Hi,
>> >
>> > I have a few questions about TextTrack behavior for modified cues and
>> > in-band tracks. Please forgive me if some of these answers are in the
spec
>> > and I didn't happen to find them. I did look. :)
>> >
>> > 1. If an attribute on a TextTrackCue like id or startTime is modified,
is
>> > this cue supposed to be internally identified in the UA as "the cue
formerly
>> > known as Y in the original source file"?
>>
>> Yes, that's how I'm reading it.
>>
>> > In the case of out-of-band text tracks this seems like a non-issue
because
>> > the cues are inserted into the TextTrackList only at the beginning of
>> > playback when they are parsed. The parser visits the source data once
and
>> > inserts cues only once.
>> >
>> > In the case of in-band tracks it is not clear what should be done
because
>> > seeking back to an earlier portion of the clip would cause the file
format
>> > parser to parse the cue data again and potentially try to insert it
again.
>> > If the original cue was modified, then the cue data in the file and the
>> > values in the existing TextTrackCue wouldn't match so simple duplicate
>> > detection would not work.
>>
>> You're correct, the case of seeking is not covered in
>>
http://www.w3.org/html/wg/drafts/html/master/embedded-content-0.html#sourcing-in-band-text-tracks
>> . It simply says "Populate the new text track's list of cues with the
>> cues parsed so far, following the guidelines for exposing cues, and
>> begin updating it dynamically as necessary."
>>
>>
>> > Is the UA expected to keep a unique internal id for each in-band
>> > TextTrackCue it creates so that it doesn't reinsert a cue that was
inserted
>> > previously, but was modified?
>>
>> I think we have to consider what the expectation of the user (and Web
>> dev in this case) is and I think that if a cue was pulled in from an
>> in-band track and then changed, that change needs to be retained from
>> a user POV. It's possible we could expect the Web dev to keep track of
>> this and have to reapply the change when seeking happens. But if we
>> want to make it easy for the Web dev, we do it in the browser.
>>
>> I think that would have two consequences:
>> * a changed cue has to remain in the browser's cache and can't be
>> dropped to free space for loading other data
>> * some means of identifying the original cue has to be retained
>>
>> What you suggested with a unique internal id makes sense to me to
>> resolve the second issue.
>>
>
> Ok. This seems like the least surprising behavior to me, but is
definitely not clear in the spec.
>
>>
>>
>> > I couldn't find anything in the HTML5 spec that clarified this
situation.
>>
>> It seems we will need to add something to this effect unless we expect
>> the Web dev to keep track of this.
>>
>
> Agreed. The spec definitely needs to be updated to clarify this.
>
>>
>>
>> > 2. When using the Media Source Extensions, it is possible to append
data
>> > over the top of previously appended data to replace it. If such an
append
>> > occurs that triggers the removal of a cue from the SourceBuffer then
it's
>> > pretty clear that the TextTrackCue can be removed from the
TextTrackList if
>> > it hasn't been modified. If the TextTrackCue has been modified though,
>> > should the UA remove it because the underlying cue in the SourceBuffer
was
>> > removed, or should it leave the modified cue in the TextTrackList?
>> >
>> > My assumption is that unmodified cues should be removed, but modified
ones
>> > should stay, but I just wanted to double check this assumption.
>>
>> Again, I wonder what the user's expectation would be here. Does the
>> replacing of previously appended data only refer to audio and video
>> data or also to text track data? If so, I think removing the modified
>> cue on the text track makes sense. If, however, you're just exchanging
>> the audio and video blocks, the cues should be retained.
>>
>
> This does not apply only to audio and video data. Text track data can be
replaced as well. It isn't clear to me that removing modified cues is the
"least surprising" option here. It is hard to determine what the
application's reasons were for modifying the cue especially if it changes
the timestamps. It might be surprising if the modified cue gets removed as
a result of an append.

Applications might modify cues to fix up wrong timings, duplicate
identifiers, typos in the content, positioning of the cue. So, anything can
really be changed.

Before the append, there has to be a remove, right? I would expect that the
remove applies to all cues in that time segment, including changed ones.
Then the append brings the original cues back and the dev will apply all
fixing-up logic to these newly appended cues "again". I don't see a problem
with this.

We have to look at cues that don't any more apply to the removed time range
though. I can think of cases where the app might want to keep them (e.g.
because the newly loaded text track segment doesn't contain them any more)
and where they should be removed (e.g. because they got updated). So, I
think we just need to make a decision on which way to go and be consistent.

E.g. if we remove cues, the app can work around it be copying cues rather
than modifying them. Or if we keep them, the app can remove them.

>>
>>
>> > 3. If we have an append that removes a cue and then another one that
adds
>> > back the cue, then the proper behavior would be to resolve the first
append
>> > according to the answer to #2 and then insert a new TextTrackCue for
the
>> > second append since there is no way to know for sure whether it was
related
>> > to the original cue. Does this sound reasonable?
>>
>> Would that not possibly result in doubly appending the same cue, even
>> if the first one was changed?
>
>
> If the rule is to only remove unmodified cues, then there isn't a
possibility for the same cue showing up twice since the first append would
remove it and the second one would add it back.

Right but then we might have the modified and the unmodified, which may
still be a duplicate (e.g. only with corrected typos).

>>
>>
>> I wonder: since the change to the original TextTrackCue was done by
>> the Web dev, maybe we can just leave the keeping track of the changes
>> to the Web dev. With these consequences:
>> * when seeking on an in-band track, the Web dev has to expect changed
>> cues to be removed and replaced back with the original cues
>
>
> Simply seeking will cause the modified cue to be reverted?!? I think that
would be more surprising than a cue being added back I think.

Not sure...

>>
>> * when using MSE and previously appended data is replaced, the Web dev
>> has to expect changed cues to be removed and replaced with those from
>> the newly appended data
>
>
> This one is a little bit of a grey area for me since this is a situation
where the application is making its intent unclear. This could be
particularly weird if the application changed the timestamps on the cue and
then an append caused the cue to be removed eventhough the modified cue
doesn't even apply to the time region of the original cue.

Right.

>>
>>
>> I'm not sure as yet whether we should help the Web dev or not. But in
>> either case, we need to clarify the spec.
>
>
> I definitely think we should clarify the spec so at least the Web dev
knows what to expect when they do these types of things.

Right. I'd like to hear preferences of JS devs.

Silvia.

>
> Aaron
>