Re: Tech Discussions on the Multitrack Media (issue-152)

On Wed, Mar 2, 2011 at 4:43 AM, Sean Hayes <Sean.Hayes@microsoft.com> wrote:
> [Silvia] "For both of the below to work, the browser has to implement an
> extraction of the cues from the binary media resource into a TextTrack
> with TextTrackCues, which then exposes it to the browser and to
> JavaScript. Since by default only TextTracks of @kind=subtitles or
> @kind=captions are rendered and these can only contain raw text, it
> probably makes sense to declare both the content advisory descriptor
> and the image-based caption as @kind=metadata."
>
> I disagree, while they may carried as text resources, there is no reason or statement that captions and subtitles cannot encode images, or indeed for an in-band track, a binary format like 608 or its SCC text-file equivalent.  SMPTE-TT is a text format but it can carry encoded images of this form (as well as the Unicode text representation), the image forms part of the caption, so labeling this as metadata is incorrect.


I was discussing how this data would be exposed through the existing
TextTrack API. From the POV of that API, data in a cue would be
exposed as exactly the text data that it is. Thus, if you expose a
in-band track that contains a base-64 encoded image as a
@kind=subtitles or @kind=captions TextTrack, the browser will display
the base-64 codes on top of the video as though it was text.
Therefore, it is required to be exposed as @kind=metadata such that it
can be exposed to JavaScript and the Web developer can write a bit of
JS code to transcode it back to binary and display in a custom manner.


>
> [Silvia] "In the case of image-based captions, I am uncertain how the images can
> actually be exposed to the browser. "
> The track format parser would know what to do with them, and would expose them with the getCueasHTML() method.

This is not the way in which the spec is currently written. Even if
the browser adds a <img> element into the cue, there remain unsolved
questions:
1. what is the url that you are going to point the <img> element to
for the image resource?
2. A @kind=captions track is automatically rendered as text on top of
the video. Thus, IIUC you will not get an interpreted <img> element,
but a verbatim <img> element displayed on top of the video.

Cheers,
Silvia.

Received on Tuesday, 1 March 2011 21:14:29 UTC