Re: timing model of the media resource in HTML5

On Feb 1, 2010, at 4:19 AM, Silvia Pfeiffer wrote:

> On Fri, Jan 29, 2010 at 12:39 AM, Philip Jägenstedt <philipj@opera.com> wrote:
> 
>> On Wed, 27 Jan 2010 12:57:51 +0100, Silvia Pfeiffer
>> <silviapfeiffer1@gmail.com> wrote:
>> 
>>> Eric further said:
>>>> 
>>>> It seems to me that because it will require
>>>> new specialized tools get the information, and because it will be really
>>>> difficult to do correctly (ten digit serial numbers?), people are likely
>>>> to just skip it completely.
>>> 
>>> There is a need for addressing the track in a unique way, i.e.
>>> javascript needs to be able to tell the media framework exactly which
>>> track it is talking about (e.g. to turn it on or off).
>> 
>> The API for exposing tracks should simply have something like .enable() for
>> each track object (or similar), there's no need to expose unique IDs to do
>> this.
> 
> How would you identify a track if you do not have a unique ID? what
> would .enable() work on if not a track identifier?
> video.firstTrack().enable() ? Even this implies that there is an order
> and that we know what is in the track.
> 
  I am not saying that tracks should not have a unique identifier, just that they should not be required in the markup. I suspect that every media container format has some kind of unique identifier for tracks, but the form of the identifier is often format specific so I don't think it is appropriate to require it in markup.  

  Some formats expose other track attributes that an author might want to use when determining what to enable, eg. media type, language, etc, so I think a DOM API is the right way to provide this. See below for more thoughts on this.

> 
>> It might be needed if we want to have e.g. <video tracks="video0,audio3"> or
>> similar. Media Fragments URI is supposed to provide a syntax for addressing
>> individual tracks, perhaps we can hook into that at some level?
> 
> Indeed the only means for addressing tracks uniquely in Media
> Fragments URI for Ogg files has so far been the serial number of the
> track. This is why I am trying to introduce track names into Ogg, so
> we have a more readable means of addressing tracks. But from a machine
> processing POV only the serial numbers are guaranteed to be uniquely
> identifying.
> 
> There was a suggestion that it would be possible to number through the
> tracks in an Ogg file based on their order given buy the serial
> number. This might be possible, but introduce an additional source of
> errors: what if an additional track is added to the Ogg file and that
> track (since its serial number is random) overthrows the order of the
> tracks. That will cause a lot more errors to happen than if we just
> used the serial numbers directly.
> 
> I read that MPEG numbers its tracks through, but if somebody knows
> exactly how it works with MPEG, that would be an interesting
> contribution. From what I can tell, the tracks are just in a sequence,
> so if you remove a track in the middle or add one, the addressing
> doesn't stay persistent either. But I may be wrong and would really
> like some input here. Is it possible to address something like
> video[0] and captions[0]?
> 
  Indeed, track order (index) is significant in MPEG-4 and QuickTime files. For example, a visual track's index determines when it is rendered so order is quite significant in a movie with multiple visual tracks. You are correct that track index changes if tracks are removed or added, but MPEG-4 and QuickTime tracks also have a persistent ID.


> 
>>> Incidentally, we do need to develop the javascript API for exposing
>>> the video's tracks no matter whether we do it in declarative syntax or
>>> not. Here's a start at a proposal for this (obviously inspired by the
>>> markup):
>>> 
>>>  video.numberTracks(); -> return number of available tracks
>>>  video.firstTrack(); -> returns first track ("first" to be defined -
>>> e.g. there is no inherent order in Ogg)
>>>  video.lastTrack(); -> returns last track ("last" to be defined)
>>>  track.next(); -> returns next track in list
>>>  track has the following attributes: type, ref, lang, role, media
>>> (and the usual contenders, e.g. id, style)
>> 
>> Yes, we need something like this.
> 
> OK, so if we cannot right now agree to have actual declarative syntax
> for it, could we for the moment focus on developing that API? While
> implementing this API, we will at least find out its flaws and we will
> also be able to exactly measure how much time and bandwidth is used in
> comparison to having declarative syntax provide this information.
> 
> 
  Instead of having methods like numberTracks and firstTrack, I think we will want to define a Track object and have a collection of Tracks on HTMLMediaElement. I haven't thought about this much, but I was imagining something like this:

interface MediaTrack {
  readonly attribute DOMString title;
  readonly attribute DOMString type;
  readonly attribute DOMString language;
           attribute boolean enabled;
  ...
};

interface MediaTracks {
  readonly attribute unsigned long length;
  getter MediaTrack item(in unsigned long index);
  ...
};

interface HTMLMediaElement : HTMLElement {
  ...
  readonly attribute MediaTracks tracks;
  ...
};


  This will allow an author to enable tracks based on whatever characteristic is important to their use case,

  if (video.tracks[1].type == "Caption") video.tracks[1].enbled = true;

or 

  if (video.tracks[1].language == "fr") video.tracks[1].enbled = true;

or whatever. 


eric