Re: Resolving TextTrackCue issues

On Mon, Sep 2, 2013 at 2:20 AM, Philip Jägenstedt <philipj@opera.com> wrote:

> On Sat, Aug 31, 2013 at 9:26 AM, Silvia Pfeiffer
> <silviapfeiffer1@gmail.com> wrote:
> > Hi all,
> >
> > Recent changes to the TextTrackCue interface had led to a fork with
> > the WHATWG spec [1] when resolving bug 21851 [2].
> >
> > This caused extensive discussion on blink-dev [3] when an intent to
> > implement was proposed.
> >
> > In the W3C WG we recognize the need for a generic cue interface type
> > with a constructor and a text attribute. It allows browsers to expose
> > cues in text tracks of video or audio files for which browsers don't
> > intend to implement parsers. It also allows JavaScript developers to
> > create time-synchronized data for media elements in any format they
> > require.
> >
> > The discussion on blink-dev exposed that the currently specified
> > solution of bug 21851 [2] in the HTML5 spec is flawed in several ways:
> >
> > (1) TextTrackCue objects that are not fully abstract create hard to
> > debug issues of backwards compatibility due to existing code that
> > assumes "new TextTrackCue()" constructs a cue with VTT semantics;
> > (2) in order to transition old TextTrackCue interface usage to "new
> > VTTCue()", it is better to remove the existing TextTrackCue
> > constructor causing hard failure (easily recognizable) instead of soft
> > failure (more difficult to recognize);
> > (3) the abstract TextTrackCue interface of the WHATWG is desirable for
> > extensibility to non-text-based cue interfaces of the future;
> > (4) the interface fork between the WHATWG and W3C spec should be removed.
> >
> > An alternative resolution to bug 21851 [2] has previously been
> > proposed and discussed: create a new interface that has the text
> > attribute and the constructor and inherits from the abstract
> > interface.
> >
> > This will result in the following interfaces:
> >
> > interface TextTrackCue : EventTarget {
> >   readonly attribute TextTrack? track;
> >
> >            attribute DOMString id;
> >            attribute double startTime;
> >            attribute double endTime;
> >            attribute boolean pauseOnExit;
> >
> >            attribute EventHandler onenter;
> >            attribute EventHandler onexit;
> > };
> >
> > [Constructor(double startTime, double endTime, DOMString text)]
> > interface GenericCue : TextTrackCue {
> >            attribute DOMString text;
> > };
> >
> > Whether VTTCue will inherit from GenericCue or from TextTrackCue will
> > be resolved in the TextTrack CG once this change has been applied to
> > the HTML5 spec.
> >
> > It is my understanding that this proposed change resolves all the
> > above listed issues. I will therefore apply these changes next week
> > unless there are any further concerns.
> >
> > Regards,
> > Silvia (as HTML spec editor).
> >
> > [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=22903
> > [2] https://www.w3.org/Bugs/Public/show_bug.cgi?id=21851
> > [3]
> https://groups.google.com/a/chromium.org/d/msg/blink-dev/-VHGnuNNUxM/Yibbv2TgDoYJ
> >
>
> Am I correct to assume that GenericCue will never be rendered?


It would be better to say "not automatically rendered by the UA without the
intervention of JS client code". That is not to say it will "never be
rendered".


> If so,
> a format initially exposed using GenericCue can likely never be
> "upgraded" to a fully functional interface, since scripts will have
> come to assume that those formats aren't rendered by the browser.


The purpose of the GenericCue is to expose the track content (in a text
form) to JS client code, so that the JS client code can take further
action. One of those further actions may be to interpret it and translate
into some format the the JS client code wishes for rendering purposes.

I'm not sure what you mean by "upgraded" to a fully functional interface,
but one could, e.g., go on to define sub-interfaces of GenericCue, such as
VTTCue, that do add UA based rendering support, e.g., by implementing
getCueAsHTML() on top of GenericCue.


> That
> seems pretty bad to me, and a good reason for browsers to not stop
> half way in supporting a format.
>

I fail to see why this is bad at all. Rather, it seems the preferred
design. The facts are as follows:

(1) there are many and arbitrary text track formats, some of which have
well defined rendering semantics, others do not;

(2) it is desirable to have the UA render at least some of those formats,
perhaps the most commonly used formats;

(3) in case the UA does not render some format, or it is not generically
renderable in the first place, then it is reasonable to expose that format
to JS client code for further disposition, some possibilities of which
include: having JS client code translate to a renderable form, e.g., for a
non-common format for which native UA support does or likely will not
appear, or in case of metadata not intended to be rendered, taking action
as appropriate for the JS client/application in hand;


> In the blink-dev thread Silvia made reference to TextTrackCueGeneric,
> [1] so it would be nice to hear from Apple (CC:ed) about whether or
> not GenericCue would be a suitable replacement. AFAICT they don't seem
> very similar, as TextTrackCueGeneric has rendering information
> attached (e.g. setFontSize) and presumably can be rendered.
>

I would also like to hear from Eric/Jer on this point, but from my
evaluation of TextTrackCueGeneric, the implementation of certain style
properties seemed to be more of a side effect of (1) having this type
inherit from the existing VTTCue laden rendering semantics of TextTrackCue
(i.e., as currently implemented in WK), and (2) possible desire to support
the ingestion of a renderable text cue based on an attributed string, e.g.,
see

https://developer.apple.com/library/ios/DOCUMENTATION/Cocoa/Reference/Foundation/Classes/NSAttributedString_Class/Reference/Reference.html

which allows one to associate font and other style data with substrings in
a string.


>
> Are there any browser vendors who support this change and want to
> expose any in-principle-renderable format using GenericCue?
>
> Also CC Hixie, since removing the spec fork was one of the goals.
>
> Philip
>
> [1]
> https://github.com/WebKit/webkit/blob/master/Source/WebCore/html/track/TextTrackCueGeneric.h
>
>