Re: Resolving TextTrackCue issues

On Sep 3, 2013, at 4:00 PM, Silvia Pfeiffer <silviapfeiffer1@gmail.com> wrote:

> On Wed, Sep 4, 2013 at 2:14 AM, Jer Noble <jer.noble@apple.com> wrote:
>> 
>> On Sep 3, 2013, at 8:46 AM, Glenn Adams <glenn@skynav.com> wrote:
>> 
>> 
>> 
>> On Tue, Sep 3, 2013 at 9:41 AM, Jer Noble <jer.noble@apple.com> wrote:
>>> 
>>> 
>>> On Sep 2, 2013, at 8:20 AM, Glenn Adams <glenn@skynav.com> wrote:
>>> 
>>> On Mon, Sep 2, 2013 at 2:20 AM, Philip Jägenstedt <philipj@opera.com>
>>> wrote:
>>>> 
>>>> 
>>>> 
>>>> In the blink-dev thread Silvia made reference to TextTrackCueGeneric,
>>>> 
>>>> [1] so it would be nice to hear from Apple (CC:ed) about whether or
>>>> not GenericCue would be a suitable replacement. AFAICT they don't seem
>>>> very similar, as TextTrackCueGeneric has rendering information
>>>> attached (e.g. setFontSize) and presumably can be rendered.
>>> 
>>> 
>>> I would also like to hear from Eric/Jer on this point, but from my
>>> evaluation of TextTrackCueGeneric, the implementation of certain style
>>> properties seemed to be more of a side effect of (1) having this type
>>> inherit from the existing VTTCue laden rendering semantics of TextTrackCue
>>> (i.e., as currently implemented in WK), and (2) possible desire to support
>>> the ingestion of a renderable text cue based on an attributed string, e.g.,
>>> see
>>> 
>>> 
>>> https://developer.apple.com/library/ios/DOCUMENTATION/Cocoa/Reference/Foundation/Classes/NSAttributedString_Class/Reference/Reference.html
>>> 
>>> which allows one to associate font and other style data with substrings in
>>> a string.
>>> 
>>> 
>>> Indeed.  Our media engine delivers styled NSAttributedStrings strings
>>> representing cues from in-band caption and subtitle tracks.  As such,
>>> GenericCue would not be a suitable replacement as all the style embedded in
>>> those cue strings would be lost, or at least would be inaccessable to
>>> script.
>> 
>> 
>> In other words, your implementation of TextTrackCueGeneric assumes these are
>> (1) renderable cues, and (2) to be rendered by UA, not script. Yes?
>> 
>> 
>> Yes.
>> 
>> How are these being used at present? In other words, what format are you
>> parsing to create attributed strings from which these renderable
>> TextTrackCueGeneric cues are created and rendered (by UA)?
>> 
>> 
>> Currently, every subtitle and caption format supported by our media engine
>> will result in cue data being delievered as NSAttributedStrings.
>> 
>> Could (should) such format(s) have a distinct public interface type exposed
>> in IDL (to script)?
>> 
>> 
>> Could? Probably.
>> 
>> Should we create distinct interface types for every format our (and every
>> other) media engine supports now and in the future?  Probably not. :)
> 
> Do you expose the existance of these in-band captions somehow to JS?
> 
> I'm concerned that if the browser renders captions automatically on
> top of video without the JS developer being able to find out about it,
> how would the JS developer know that there are captions and to avoid
> rendering another lot themselves - or rendering something else in the
> space of the captions?
> 
  On versions of the OS where it is possible for WebKit to "take over" rendering of in-band captions from the media engine, they behave just like out-of-band tracks: in-band tracks are part of the video.textTracks and the cues are part of track.cues/activeCues (when appropriate).

  On versions of the OS where the system frameworks do not have the necessary API to override cue rendering, in-band tracks are part of video.tracks so they can be enabled/disabled by script but cues are rendered by the media engine.

eric