Re: Tech Discussions on the Multitrack Media (issue-152)

On Wed, 16 Feb 2011 03:31:47 +0100, Silvia Pfeiffer  
<> wrote:

> On Wed, Feb 16, 2011 at 12:08 PM, Jonas Sicking <> wrote:
>> On Tue, Feb 15, 2011 at 4:19 PM, Silvia Pfeiffer
>> <> wrote:
>>> On Wed, Feb 16, 2011 at 5:36 AM, Mark Watson <>  
>>> wrote:
>>>> Hi Philip,
>>>> Just a quick note that the "alternative" vs "additional" distinction  
>>>> is not always completely clear. Video with different camera angles  
>>>> (gimmiky or not) could be considered as an alternative, or could be  
>>>> rendered as picture-in-picture, or multiple thumbnail videos could  
>>>> show beside the main video (some sports sites already do this kind of  
>>>> thing).

Sure, but all of those modes should be achieved by the author making it  
happen with CSS. At the risk of making a strawman argument, I honestly  
can't see browsers allowing the user to change the rendering of the page  
to achieve PiP or something like that when the author hasn't provided for  
it, messing with the layout like that seems both weird and unlikely to be  
useful. Of course we can have User JavaScript and User CSS to do that kind  
of thing, though.

>>>> If you have the output capabilities (e.g. wireless headphones plus  
>>>> regular speakers) then simultaneous output of different audio  
>>>> languages might make sense.

Sure, audio tracks are quite simple. Because they don't have visual  
rendering (except <audio controls>) it doesn't matter much which the  
solution is on the syntax level -- there would be no noticeable difference  
between overloading <track>, nested <audio> or external <audio>.

>>>> Not to say these are compelling use-cases, just that the markup  
>>>> should indicate what the media actually is such that the player can  
>>>> decide what to do, without any hard "alternative" vs "additional"  
>>>> distinction.
>>> I think I generally agree.
>> Is there anything that prevents an implementation from displaying
>> multiple "alternative" streams at the same time? Even if they are
>> explicitly labelled and in spec called "alternative" I wouldn't think
>> there is. Compare to how there are currently alternative CSS
>> stylesheets. There is nothing preventing an implementation from
>> displaying multiple windows which have different alternative
>> stylesheets applied, as long as the DOM acts as though a specific one
>> is applied (you can only return one value for .offsetTop). This is a
>> model that works quite well IMHO since it doesn't require page authors
>> to keep more exotic UAs in mind, allowing UAs to freely experiment.
> I think it all burns down to what the user/UA select to activate. If
> there are multiple tracks that should really be alternative but are
> presented together, then it's up to the user to deactivate the one
> they don't want to see/hear. Even with <track> elements there is
> nothing prohibiting multiple tracks from being active at the same
> time. It's up to the UA to present these and up to the user to decide
> if that is the presentation they would like to see. So, I don't
> actually think we need to put an attribute on the media track for
> alternative/additional, since the UA will not actually use it as input
> I used to think that alternative/additional was a big thing, too, and
> it made sense to tell the UA about it so it can act appropriately, but
> it becomes very complex very quickly: which group of tracks is
> alternative to which other (e.g. all the English tracks a+v+t plus
> original video against all the German tracks plus original video?) and
> the logic becomes almost impossible to represent, and even worse to
> interpret. So, I would think it's easiest if the browser just tries to
> display them and leaves the choice of managing the active tracks to
> the user.

What I think the browser shouldn't do is do fancy things with the layout  
of extra video tracks apart from applying the page CSS. This means that  
alternative and additional video tracks are in fact very different.

An alternative video track should be rendered in the <video> content box,  
completely replacing the original video, which need not even be decoded.

An additional video track must have a layout box outside of the master  
<video>, unless we want to limit ourselves to just overlay tracks, in  
which case we'd also need to have more special handling for rendering  
child <video>/<track> elements.

I don't want a solution where the UA has no idea about how to render  
additional video tracks and is forced to confront the user with a complex  
UI allowing the user to manipulate the layout in ways the page author  
hadn't thought of.

In short: For audio tracks, a solution similar to <track> would be OK, but  
for video tracks it would not. I would only consider option 6 or 7 for  
video tracks.

Philip Jägenstedt
Core Developer
Opera Software

Received on Wednesday, 16 February 2011 08:02:52 UTC