RE: Tech Discussions on the Multitrack Media (issue-152)

That behavior is dependent on the parser allotted to the text track; which depends on the type of caption data. Since <track> is format independent, then it would not render the image as base 64 text if the format parser knows what that means, as SMPTE-TT does. I believe you are conflating the general text track behavior with a specific  text track parser.

"1. what is the url that you are going to point the <img> element to
for the image resource?"
For example:
data:image/png;base64,iVBORw0KGg....

"2. A @kind=captions track is automatically rendered as text on top of
the video. Thus, IIUC you will not get an interpreted <img> element,
but a verbatim <img> element displayed on top of the video."

>From the spec: "Associated with the list are the rules for updating the text track rendering appropriate for the format in question; for WebVTT, this is the rules for updating the display of WebVTT text tracks. "

The rules for updating are specific to each track format. By implication, for something other than WebVTT, the rules will be whatever that format requires.

Thanks,
Sean.

-----Original Message-----
From: Silvia Pfeiffer [mailto:silviapfeiffer1@gmail.com] 
Sent: 01 March 2011 21:14
To: Sean Hayes
Cc: David Singer; Bob Lund; public-html@w3.org
Subject: Re: Tech Discussions on the Multitrack Media (issue-152)

On Wed, Mar 2, 2011 at 4:43 AM, Sean Hayes <Sean.Hayes@microsoft.com> wrote:
> [Silvia] "For both of the below to work, the browser has to implement an
> extraction of the cues from the binary media resource into a TextTrack
> with TextTrackCues, which then exposes it to the browser and to
> JavaScript. Since by default only TextTracks of @kind=subtitles or
> @kind=captions are rendered and these can only contain raw text, it
> probably makes sense to declare both the content advisory descriptor
> and the image-based caption as @kind=metadata."
>
> I disagree, while they may carried as text resources, there is no reason or statement that captions and subtitles cannot encode images, or indeed for an in-band track, a binary format like 608 or its SCC text-file equivalent.  SMPTE-TT is a text format but it can carry encoded images of this form (as well as the Unicode text representation), the image forms part of the caption, so labeling this as metadata is incorrect.


I was discussing how this data would be exposed through the existing
TextTrack API. From the POV of that API, data in a cue would be
exposed as exactly the text data that it is. Thus, if you expose a
in-band track that contains a base-64 encoded image as a
@kind=subtitles or @kind=captions TextTrack, the browser will display
the base-64 codes on top of the video as though it was text.
Therefore, it is required to be exposed as @kind=metadata such that it
can be exposed to JavaScript and the Web developer can write a bit of
JS code to transcode it back to binary and display in a custom manner.


>
> [Silvia] "In the case of image-based captions, I am uncertain how the images can
> actually be exposed to the browser. "
> The track format parser would know what to do with them, and would expose them with the getCueasHTML() method.

This is not the way in which the spec is currently written. Even if
the browser adds a <img> element into the cue, there remain unsolved
questions:
1. what is the url that you are going to point the <img> element to
for the image resource?
2. A @kind=captions track is automatically rendered as text on top of
the video. Thus, IIUC you will not get an interpreted <img> element,
but a verbatim <img> element displayed on top of the video.

Cheers,
Silvia.

Received on Tuesday, 1 March 2011 21:25:50 UTC