Re: Using WebVTT metadata tracks and missing API

On Tue, Nov 12, 2013 at 9:43 AM, Cyril Concolato
<cyril.concolato@telecom-paristech.fr> wrote:
> Le 11/11/2013 11:18, Silvia Pfeiffer a écrit :
>
>> On Mon, Nov 11, 2013 at 6:14 PM, Cyril Concolato
>> <cyril.concolato@telecom-paristech.fr> wrote:
>>>
>>> Le 11/11/2013 11:06, Silvia Pfeiffer a écrit :
>>>
>>>> Hi Cyril,
>>>>
>>>> On Mon, Nov 11, 2013 at 5:49 PM, Cyril Concolato
>>>> <cyril.concolato@telecom-paristech.fr> wrote:
>>>>>
>>>>> Le 30/10/2013 10:03, Silvia Pfeiffer a écrit :
>>>>>
>>>>>> BTW: We need to keep the connection between the video and the VTT
>>>>>> resource to a minimum - in particular I object to adding the mime type
>>>>>> of the media resource, or the codec, or codec paramters to the vtt
>>>>>> file header. The VTT file needs to be able to be used with different
>>>>>> encodings of the same video file through the <track> element, which
>>>>>> can relate to different <source> elements.
>>>>>>
>>>>>> So, the following header fields that you are using are actually
>>>>>> harmful: baseMediaFile, MPEG-4-streamType,
>>>>>> MPEG-4-objectTypeIndication, sampleRate, numChannels,
>>>>>> MPEG-4-DecoderSpecificInfo . Please don't start exposing such
>>>>>> information in a VTT file. This information must not be made part of a
>>>>>> VTT file - it is far too easy to get out of sync with an actual video
>>>>>> file and is already out of sync with a different encoding of the same
>>>>>> video file. The VTT file is not a companion file to a specific media
>>>>>> resource. If you need to expose such information in a file, I
>>>>>> recommend writing a separate file, e.g. video_metadata.xml.
>>>>>
>>>>> You misunderstood me. The information is not duplicated in the VTT
>>>>> file,
>>>>> it
>>>>> is *only* in the VTT file. In this example, I assume that some
>>>>> audio/video/other data carried in some original media container (say
>>>>> mp4)
>>>>> is
>>>>> not supported but the browser's demultiplexing stack and thus not
>>>>> exposed
>>>>> to
>>>>> JS API via a VideoTrack or AudioTrack interface. So to work around that
>>>>> limitation, I'm thinking about using WebVTT to expose the data (or a
>>>>> reference to the data). In the example, I'm putting the information
>>>>> that
>>>>> was
>>>>> initially in the MP4 file in the WebVTT file: some in the header (from
>>>>> MP4
>>>>> header to VTT header), some in the cue data (from MP4 samples to VTT
>>>>> cues).
>>>>
>>>> What is a JS developer supposed to do with this information?
>>>
>>> Parse and process it according to the format (the JS developer knows how
>>> it
>>> the WebVTT describes enough information) possibly using Web Audio API,
>>> Canvas, WebGL, SVG, ...
>>
>> baseMediaFile, MPEG-4-streamType, MPEG-4-objectTypeIndication,
>> sampleRate, numChannels, MPEG-4-DecoderSpecificInfo all sound very
>> much like header fields of a media stream. You're not suggesting SW
>> decoding of tracks?
>
> Yes, I am. It might not make sense for all track types, on all devices, but
> for some it does (e.g. 3d graphics, motion jpeg, depth tracks, ...).

Right, so you are going to parse the MP4 file again and starting to
decode the file anyway, which includes parsing out that information
again. Then there is no need to duplicate this information into a text
track. All you really need is an indication to the Web page that there
is a file that it needs to decode. Are you expecting the browser to
expose this information even when it doesn't support decoding the
track? Do you have a concrete example? I am still grappling with the
use case here, actually, since if it's a video track or audio track,
then it should simply go into a <video><source> tag.

Silvia.

Received on Tuesday, 12 November 2013 01:52:13 UTC