Re: TextTrack API changes

On Sat, May 11, 2013 at 5:55 PM, Silvia Pfeiffer
<silviapfeiffer1@gmail.com>wrote:

> On Wed, May 8, 2013 at 1:11 PM, Glenn Adams <glenn@skynav.com> wrote:
> >
> > On Tue, May 7, 2013 at 7:52 PM, Glenn Adams <glenn@skynav.com> wrote:
> >>
> >>
> >> On Tue, May 7, 2013 at 7:44 PM, Glenn Adams <glenn@skynav.com> wrote:
> >>>
> >>> Silvia,
> >>>
> >>> At this point, I would suggest the following resolution, subject to
> >>> discussion and agreement:
> >>>
> >>> 1. leave TextTrackCue.text in place, with no name change;
>
> I agree that adding .text back onto TextTrackCue makes sense to me.
>
>
> >>> 2. move getCueAsHTML() to WebVTTCue; other formats can define an
> appropriate
> >>> mapping and necessary partial interfaces on TextTrackCue or a subclass;
>
> That's already how it is now.
>
>
> >>> 3. either (1) remove TextTrackCue constructor and define
> TextTrack.addCue
> >>> with a content type implied by the content type of the TextTrack (see
> below)
>
> That doesn't make sense to me - TextTrack.addCue(cue) adds a cue to a
> TextTrack, where the cue has been constructed before. Thus, if you
> wanted to be able to create a generic TextTrackCue, i.e. one where the
> content type has not been defined yet and is only interpreted by the
> content type of the TextTrack, then we still need a constructor for
> TextTrackCue. This was, in fact, what I proposed.
>
>
> >>> or (2) keep TextTrackCue constructor and add an optional media type
> >>> parameter;
>
> I don't think a media type parameter is necessary. The TextTrackCue
> can stand on its own as a plain vanilla text-only cue associated with
> a time interval until an interpretation is necessary. That
> interpretation is only necessary when it gets associated with a
> TextTrack through addCue().
>
>
> >> s/addCue/createCue/ since addCue is defined already meaning something
> >> other than create a cue
> >>
> >>>
> >>> 4. add an optional media type parameter to
> HTMLMediaElement.addTextTrack,
> >>> thus allowing client JS to declare the intended type of any cue
> created by
> >>> TextTrack.addCue
>
> Hmm...
>
> We are now dealing with multiple different "types" here - let's see if
> they are necessary to expose and how to expose them:
>
> * file format of text track: this is the "encapsulation" format in
> which the cues are delivered, e.g. WebVTT files, TTML files, MPEG
> files, WebM files
>   -> I don't think this has to be exposed to JS, because we abstract
> away from the file format in the TextTrack object and the browser does
> the parsing of the file for us
>
> * cue format of the text track: this is the parsing of the cues after
> extraction from the file, e.g. WebVTT caption cues (as extracted from
> WebVTT files, MPEG files, WebM files), WebVTT chapter cues, TTML
> caption cues, MPEG-specific metadata cues
>   -> in the past we have introduced the
> inBandMetadataTrackDispatchType IDL attribute to expose the type of
> data that is exposed to JS from a metadata track from inband tracks. I
> believe we are now talking about doing the same thing for other kinds
> of tracks coming from inband, <track> or even JS-created.
>
> As I don't think anyone has implemented the
> inBandMetadataTrackDispatchType IDL attribute yet (please let me know
> if there is an implementation!), we could rename this attribute to
> cueType and expose the cue type for all cues there.
>
> The cueType would be set by the browser when the TextTrack is created
> from in-band or from parsing a <track> element. In the case of WebVTT
> from WebM, this is already set to “D_WEBVTT/kind“, where kind is one
> of SUBTITLES, CAPTIONS, DESCRIPTIONS, or METADATA. The same could be
> done for WebVTT cues coming from <track>. Do you think the MPEG2 and
> MPEG4 types as described in [1] would work for this, too? Would they
> be able to expose the cue format?
>
> [1]
> http://www.w3.org/TR/html5/single-page.html#steps-to-expose-a-media-resource-specific-text-track
>
> For JS-created tracks, we'd have a default TextTrack cueType, which
> may just be plain text. That would work with any kind of track (kind
> being one of subtitle, caption, metadata, chapter, descriptions).
> However, when the first cueType is added from a cue that is not just a
> TextTrackCue, but a more specific cue such as a WebVTTCue, then the
> cueType of the TextTrack changes to that cueType and cannot be changed
> again. All TextTrackCues that are not a WebVTTCue will be rejected.
>

First, I'm talking about the Media Type of a text track resource here, not
a specific @kind (usage) of a text track resource. For example, "text/vtt",
"application/ttml+xml", "application/x-mpeg2-psi" [I just made that up],
etc.

Now, let me try to be more concrete regarding uses:

Ideally, <track> would use <source> in the same fashion as <video> and
<audio>, in order to allow use of the resource selection algorithm for
alternate track resources:

<video src="video.mp2t">
  <!-- in- or out-of-band captions, three alternative sources -->
  <track kind="captions">
    <!-- out-of-band VTT -->
    <source src="video.vtt" type="text/vtt">
    <!-- out-of-band TTML -->
    <source src="video.ttml" type="application/ttml+xml">
    <!-- in-band 708 -->
    <source src="video.mp2t" type="application/x-cea-708">
  </track>
  <!-- in-band MPEG-2 PSI, only one source -->
  <track kind="metadata" src="video.mp2t" type="application/x-mpeg2-psi" />
  <!-- out-of-band custom metadata, two alternative sources -->
  <track kind="metadata">
    <!-- out-of-band custom metadata, type 1 -->
    <source src="video.md1" type="application/x-metadata-1">
    <!-- out-of-band custom metadata, type 2, in case type 1 not supported
-->
    <source src="video.md2" type="application/x-metadata-2">
  </track>
</video>

Using this mechanism, the UA fetches track resources according to what
track media types it supports and what resources are actually resolvable.

Once it has resolved a track's alternate source references to an actual
resource (whether out-of-band or in-band), the UA determines the actual
content type of the resource (when it sniffs/parses it).

So, let's say that:

(1) HTMLSourceElement.type (or HTMLTrackElement.type) returns the advisory
(hint) author supplied type (may or may not be the resolved type); and

(2) HTLTrackElement.track.type returns the actual (sniffed/parsed) type as
determined by the UA and selected by the resource selection algorithm;

Why is this useful? Because it could help the client JS code to determine
things like:

   - what possible interface types are supported by a cue instance that the
   UA constructs for that type;
   - what possible different formats may be returned from TextTrackCue.text;

Of course, I am not discussing the case where the client JS invokes cue
constructors, but cases where the UA is constructing cues based on its
support for specific track content types. Although, if the client JS is
constructing cues (using a constructor or a TextTrack.createCue method),
then knowing the above information may help there as well.

Now, for the case where client JS wants to construct a track, then
HTMLMediaElement.addTextTrack (possibly renamed to createTextTrack) should
support an optional type parameter which is used to initialize
TextTrack.type, and subsequently, TextTrack.type is used to constrain the
type(s) of cues constructed by a TextTrack.createCue method or constrain
the type(s) of cues that can be added via TextTrack.addCue.

I also wonder, given the example I constructed above, why it is necessary
for <track> to specify a non-empty @src. In the case of an in-band track
which is intended to be demuxed from an outer container resource
(video.mp2t in the above example), I would expect no @src to be necessary,
e.g.,

  <!-- in-band MPEG-2 PSI, only one source -->
  <track kind="metadata" type="application/x-mpeg2-psi" />

instead of

  <!-- in-band MPEG-2 PSI, only one source -->
  <track kind="metadata" src="video.mp2t" type="application/x-mpeg2-psi" />

And also, the same might apply to the last <source>

    <!-- in-band 708 -->
    <source type="application/x-cea-708">

instead of

    <!-- in-band 708 -->
    <source src="video.mp2t" type="application/x-cea-708">

That is @src shouldn't be needed if it's value would be the same as
specified in an outer media element (video in this case).

If <source> children were added to <track>, then
HTMLTrackElement.currentSrc should be added as well.