Re: Proposal for Audio and Video Track Selection and Synchronisation for Media Elements

On Mon, 21 Mar 2011, Philip Jägenstedt wrote:
>
> On Mon, 21 Mar 2011 12:00:46 +0100, Ian Hickson <ian@hixie.ch> wrote:
> > 
> > * Allowing authors to use CSS for presentation control, e.g. to
> >   control where multiple video channels are to be placed relative to
> >   each other.
> 
> It's clear how to do this declaratively, but with scripts there seems to be a
> few steps involved:
> 
> 1. Create/find a second video element and position it with CSS.
> 
> 2. Binds the two video tracks together with the same MediaController using
> .mediaGroup or creating a new controller.
> 
> 3. Set the src of the second video element to the same as the first.
> 
> 4. Wait until loadedmetadata and then enable the alternative video track.

Well if you know ahead of time which track you want you can just use 
fragment identifiers in the 'src' attribute, but yes, worse case it would 
look something like this:

  var controller = new MediaController();
  var video1 = document.createElement('video');
  video1.controller = controller;
  video1.src = 'video.foo';
  video1.onloadedmetadata = function () {
    video1.videoTracks.select(1);
  };
  var video2 = document.createElement('video');
  video2.controller = controller;
  video2.src = 'video.foo';
  video2.onloadedmetadata = function () {
    video2.muted = true;
    video2.videoTracks.select(2);
  };


> This seems like it would work, but what if steps 2 and 3 happen in the 
> other order?

So long as they happen in the same task, it doesn't matter; the loading of 
the media resource happens asynchronously after the task ends.


> Is this a bug, or do we expect browsers to become clever enough to figure out
> that the same decoding pipeline can be reused without any kind of hint?

It doesn't seem that difficult to compare URLs... am I missing something?


> >    <video src="movie.vid#track=Video&amp;track=English" autoplay controls
> > mediagroup=movie></video>
> >    <video src="movie.vid#track=sign" autoplay mediagroup=movie></video>
> 
> This is using Media Fragment URI:
> http://www.w3.org/2008/WebVideo/Fragments/WD-media-fragments-spec/#naming-track
> 
> It appears that the proposal assumes that track dimension can only be
> specified one in a valid Media Fragment, but this is unfortunately not the
> case. The MF spec states that "Multiple track specification is allowed, but
> requires the specification of multiple track parameters."
> Perhaps this is not a problem, #track=Alternative&track=Commentary would just
> result in a resource with two tracks, and the first (Alternative) would be
> used. However, how should this be reflected in audioTracks and videoTracks?

This is handled in the HTML spec in the "Once enough of the media data has 
been fetched to determine the duration of the media resource, its 
dimensions, and other metadata, and once the text tracks are ready" step 
of the resource fetch algorithm (substep 9, currently).


> > RISKS
> > * It's possible that it is still too early for us to be adding any
> >   kind of multi-track feature given the current implementation
> >   priorities of user agents.
> 
> Indeed, the complexity of implementing this is significant. It requires 
> a very capable media framework to do things like gapless looping of one 
> track synchronized with another and to determine when decoding pipelines 
> can be shared and not.

If the chairs determine that there is consensus that we should only 
address a subset of the use cases today, I'd be happy to subset the 
proposal appropriately. The proposal is intended to show primarily where I 
think we should be headed, it's often the case that we get to where we're 
headed in small incremental steps. (There's a lot of stuff in the spec 
that's commented out but already mostly specced out on these grounds. For 
example, the drag-and-drop feature has a lot more features specced but 
commented out than are currently obvious from reading the spec. Even the 
proposal we're talking about here has a section commented out talking 
about automatic ducking for audio description tracks.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'