Re: Resolving TextTrackCue issues

On Thu, Sep 5, 2013 at 1:52 AM, Bob Lund <B.Lund@cablelabs.com> wrote:
>
>
> On 9/4/13 12:48 AM, "Silvia Pfeiffer" <silviapfeiffer1@gmail.com> wrote:
>
>>On Wed, Sep 4, 2013 at 2:14 PM, Eric Carlson <eric.carlson@apple.com>
>>wrote:
>>>
>>> On Sep 3, 2013, at 5:16 PM, Silvia Pfeiffer <silviapfeiffer1@gmail.com>
>>>wrote:
>>>
>>>> On Wed, Sep 4, 2013 at 9:19 AM, Eric Carlson <eric.carlson@apple.com>
>>>>wrote:
>>>>>
>>>>>> Do you expose the existance of these in-band captions somehow to JS?
>>>>>>
>>>>>> I'm concerned that if the browser renders captions automatically on
>>>>>> top of video without the JS developer being able to find out about
>>>>>>it,
>>>>>> how would the JS developer know that there are captions and to avoid
>>>>>> rendering another lot themselves - or rendering something else in the
>>>>>> space of the captions?
>>>>>
>>>>>  On versions of the OS where it is possible for WebKit to "take over"
>>>>> rendering of in-band captions from the media engine, they behave just
>>>>>like
>>>>> out-of-band tracks: in-band tracks are part of the video.textTracks
>>>>>and the
>>>>> cues are part of track.cues/activeCues (when appropriate).
>>>>
>>>> So for the JS dev they are exposed as instances of the old
>>>> TextTrackCue interface?
>>>
>>>   Yes.
>>>
>>>> The VTTCue interface sufficiently satisfies this use case then?
>>>
>>>   True, but the in-bad cues may or may not be WebVTT originally. Is
>>>there an advantage to using VTTCue instead of the old TextTrackCue
>>>interface?
>>
>>getCueAsHTML() of VTTCue interprets what's in .text as WebVTT content.
>>Exposing other format content in that way would not make much sense,
>>just as parsing HTML through a Word Doc format parser makes not much
>>sense. So, IMHO neither VTTCue nor the old TextTrackCue interface are
>>appropriate here since you're not dealing with WebVTT content
>>originally.
>>
>>
>>>> What is the content in .text ?
>>>>
>>>   The cue text.
>>
>>Just to clarify: Is that a plain text version of the original cue
>>content? For example, for CEA608 content from a SCC file it would just
>>be the plain text of the cue stripped of all the other characters? Or
>>in the case of SRT, all tags are stripped and just the plain text is
>>exposed?
>>
>>If you do that, then we're just pretending the in-band text track was
>>actually WebVTT content.
>>
>>If you don't return plain text, but the actual original content of the
>>cue, you get the wrong behaviour with getCueAsHTML() and the rendering
>>algorithm of the old TextTrackCue or the new VTTCue interface.
>>
>>
>>>>>  On versions of the OS where the system frameworks do not have the
>>>>> necessary API to override cue rendering, in-band tracks are part of
>>>>> video.tracks so they can be enabled/disabled by script but cues are
>>>>>rendered
>>>>> by the media engine.
>>>>
>>>> In this case, I assume only the existence of the track, but not of the
>>>> cues is exposed to JS? I.e. track.cues/activeCues is empty? Or are you
>>>> listing fully-abstract TextTrackCue instances here to at least provide
>>>> starttime/endtime to the JS devs?
>>>>
>>>   Correct, the cues are not available to WebKit at all in this case.
>>
>>OK. Would you consider using the abstract TextTrackCue interface of
>>the WHATWG spec for exposing these cues, so JS developers can at least
>>react to cue change events?
>>
>>
>>>> And an orthogonal question: you've probably seen the Cablelabs spec
>>>> for exposing MPEG-2 in-band text tracks of different types to HTML[1].
>>>> Seeing as both Cablelabs and the HTML spec are trying to accommodate
>>>> using in-band text tracks in HTML/JS, what do you suggest is the best
>>>> way forward to specify this?
>>>>
>>>> * Program Map Table: would you suggest to use VTTCue with
>>>> @kind=metadata, GenericCue, or a new PMTCue interface?
>>>>
>>>   It seems to me that text track content that a UA does not render
>>>itself is, by definition, metadata.
>>
>>In this case, that definition works, because no @kind value matches
>>the semantic content of a PMT track.
>>But caption data that is not rendered is still semantically of
>>@kind=captions.
>>
>>> Again, it is not WebVTT data so is there an advantage to VTTCue versus
>>>old TextTrackCue interface?
>>
>>Not between those two, since they are identical.
>>
>>
>>>> * CEA708 track: assuming we don't want to introduce CEA708Cue, how
>>>> would that best be supported?
>>>>
>>>   Are any browsers planning to support 708 captions natively?
>>
>>What does "support" mean?
>>Parse them out of a MPEG-2 file (like the Cablelabs spec suggests) and
>>expose them to JS?
>>Or.. Rendering them like you do for formats that the QuickTime
>>framework already supports, but without exposing the original data and
>>pretending it's WebVTT?
>>Or .. go all the way to exposing the format with its own features?
>>
>>Only the 3rd option requires specification of a new interface. The
>>first option requires something like the GenericCue interface, since
>>we need to give the original content of the cue to JS to parse.
>>
>>It seems that Cablelabs expected that there would be browsers that
>>implement parsing, but not rendering for text tracks in MPEG-2. It
>>would be good if there was a statement from browser vendors (or set
>>top box developers that use browser rendering engines or the like) if
>>that is a realistic use case.
>

> We've implemented UA rendering of 708 captions - this is done by
> converting 708 to WebVTT and then taking advantage of the existing WebVTT
> rendering.

Out of curiosity: In the .text attribute of VTTCue / the old
TextTrackCue interface, are you exposing the 708 cue content or the
converted WebVTT cue content?


> The spec calls for making the caption data from the MPEG-2 TS
> available to JS via .text.

Right. Is that on a track with @kind=captions or @kind=metadata?

Thanks for your input!
Silvia.

Received on Thursday, 5 September 2013 14:23:40 UTC