Re: Proposal for Audio and Video Track Selection and Synchronisation for Media Elements

On Tue, 22 Mar 2011, Philip Jägenstedt wrote:
> > 
> > Well if you know ahead of time which track you want you can just use
> > fragment identifiers in the 'src' attribute, but yes, worse case it would
> > look something like this:
> > 
> >  var controller = new MediaController();
> >  var video1 = document.createElement('video');
> >  video1.controller = controller;
> >  video1.src = 'video.foo';
> >  video1.onloadedmetadata = function () {
> >    video1.videoTracks.select(1);
> >  };
> >  var video2 = document.createElement('video');
> >  video2.controller = controller;
> >  video2.src = 'video.foo';
> >  video2.onloadedmetadata = function () {
> >    video2.muted = true;
> >    video2.videoTracks.select(2);
> >  };
> > 
> > 
> > > This seems like it would work, but what if steps 2 and 3 happen in the
> > > other order?
> > 
> > So long as they happen in the same task, it doesn't matter; the loading of
> > the media resource happens asynchronously after the task ends.
> 
> It's inevitable that someone will set video2.controller in
> video2.onloadedmetadata.

Sure, but by that point they have much bigger problems (e.g. the videos 
will be playing out of sync).

(Note that the same possible mistake can be made with all the proposals on 
the table that rely on a controller, whether that's a separate object as 
in my proposal or a asymetric media element as in two of the other 
proposals.)


> > > Is this a bug, or do we expect browsers to become clever enough to 
> > > figure out that the same decoding pipeline can be reused without any 
> > > kind of hint?
> > 
> > It doesn't seem that difficult to compare URLs... am I missing 
> > something?
> 
> That makes it easy to reuse the network connection and cache, Opera 
> already does this. What I'm talking about here is using a single 
> decoding pipeline when showing multiple video streams of same resource.

As am I. As far as I can tell, my point stands.


> In your proposal, the decoding of two video streams from the same resource is
> not as tightly coupled, one can only make educated guesses.

There's no guesswork. You know it's the same resource, and you know 
they're being synchronised. What is there to guess about?


> If one initially does share a single demuxer, it might become necessary 
> to split that into two if the controller or playbackRate of one element 
> changes.

Or if either is offset from the other, sure. That's the case with any 
proposal that addresses all the use cases given in these CPs:

   http://lists.w3.org/Archives/Public/public-html/2011Mar/0436.html
   http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Change_Proposal
   http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Change_Proposal_2

...which seems to be all of the proposals except Frank's, which is much 
more limited in scope.


> I'm not saying that this is necessarily a bad thing, I just want to make 
> sure that everyone is fully aware of the complexity. I think there are a 
> few sane ways of going about this:
> 
> * Only allow setting the controller of an audio/video element while 
> readyState == HAVE_NOTHING, so that the browser knows when starting to 
> fetch a resource whether or not it's a good idea to share the decoding 
> pipeline between two elements.

That doesn't help for the case where the tracks start in sync and the 
author suddenly advances one so it's offset, or changes the playback rate 
of a single track.


> * Make the connection more explicit by letting each video track expose a 
> Stream object which could be set to the .src of another video element. 
> The other video element would not be able to change the playbackRate, so 
> it would always be safe to reuse a single demuxer.

We could do that, certainly. It seems a bit weird in that it makes a 
<video> into a feature with two orthogonal purposes, but I suppose that 
might be ok. (Maybe alternatively we could have a way to get a stream back 
from an XMLHttpRequest and we could use that, it would make more sense.)

However, I'm not sure it really is any better than comparing the URL, 
which you already have to do anyway to share the download in cases such as 
this in other parts of the platform -- and once you're sharing the 
download, sharing the decoding pipeline seems like an obvious move.


> * Be explicit about what is expected of implementations and add to the 
> best practices for implementors of media elements that they are expected 
> to be able to merge and split decoding pipelines of separate media 
> elements while they are playing in order to implement multitrack in a 
> non-wasteful manner.

I'd be happy to do that, sure.

Filed as: http://www.w3.org/Bugs/Public/show_bug.cgi?id=12367


> OK, for the record I question the necessity of:
> 
> * Synchronized playback of tracks of the same resource at different 
> playbackRates and offsets.
>
> * Synchronized playback of looping resources.

From the spec point of view support for these is almost free, so I'm 
reluctant to drop support for them unless it's really hard to implement. 
From the implementation point of view, it seems that there's nothing 
especially different about supporting that than supporting synchronisation 
of separate files. Could you elaborate on this point?

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'