RE: How do we enable/select tracks for playout?

Wait a minute... what if primaryVideoTrack was only valid locally?  (In other words, it wouldn't be signaled to the other side over a PeerConnection.)  It would still be useful to let the application define the Track to display, rather than letting the sink guess.  In this case, changing the value of primaryVideoTrack would cause the sink to change which track it was displaying.  So Peer1 sends multiple video streams to Peer2, and the code on Peer2's side uses primaryVideoTrack to toggle which Track its local video element displays. Peer1 wouldn't know about this, but that should be ok. 

- Jim

-----Original Message-----
From: Jim Barnett [mailto:Jim.Barnett@genesyslab.com] 
Sent: Thursday, March 21, 2013 9:47 AM
To: Stefan Håkansson LK
Cc: public-media-capture@w3.org
Subject: RE: How do we enable/select tracks for playout?

Stefan,
  It would certainly be simpler - for implementers - to leave the selection of the video track up to the sink or to let it be random.  This would lead to indeterminacy in applications, and to possible non-portability of apps (in the sense that a given app might work well with Firefox's random choice, but not Chrome's).

One possibility would be to leave it out for now, and add it later if non-portability turns out to be a practical, and not just hypothetical, problem.  

- Jim

-----Original Message-----
From: Stefan Håkansson LK [mailto:stefan.lk.hakansson@ericsson.com]
Sent: Thursday, March 21, 2013 9:39 AM
To: Jim Barnett
Cc: public-media-capture@w3.org
Subject: Re: How do we enable/select tracks for playout?

On 3/21/13 1:30 PM, Jim Barnett wrote:
>> For the audio part it is pretty clear - all tracks that are "enabled" 
>> should be used (and mixed) for playout. Video is less so.
>> If there is more than one video track that is "enabled", which one
>> >should be selected? There is no "first".
>
> This was the idea behind adding some way of extending the API to allow 
> setting a primaryVideoTrack.  Setting or changing this value would not 
> produce any difference in behavior in the MediaStream or Track (if the 
> Track is muted, it stays muted), but would serve as an indication to 
> the sink to treat the Track as the 'first'.

What would the consequences be if we did not add this, but instead let the track to select be random?

I'm asking only because I can see two issues with a "primaryVideoTrack":

1. In the networked case, it is one more thing we need to convey to the other side. Knowing all the problems we have with the signaling I would like to avoid adding things...

2. If we have a "primaryVideoTrack" setting that can be associated with one video track in a MediaStream, then it is not far fetched that this setting can be changed (to indicate another track as "primary") in operation - and then you could in principle have one track being "primary" in the MediaStream and another being "selected" in the sink (media element). I think that would be confusing.

Br,
Stefan


>
> - Jim -----Original Message----- From: Stefan Håkansson LK 
> [mailto:stefan.lk.hakansson@ericsson.com] Sent: Thursday, March 21,
> 2013 6:24 AM To: public-media-capture@w3.org Subject: Re: How do we 
> enable/select tracks for playout?
>
> It seems to me that we are converging on:
>
> * Enable/disable of a MediaStreamTrack is something different from 
> selecting which video track that is being played in a video element * 
> Those should be separate things (you can select a disabled track for
> playout)
>
> This basically means that we don't need to change any of what has 
> already been defined: * MediaStreamTrack keeps its "enabled"
> attribute * media elements allow for selecting a specific video track 
> for playout
>
> The one outstanding question is how to deal with the case of several 
> enabled video tracks in a MediaStream when being attached to a video 
> element. Step 11 of the "resource fetch algorithm" of the media 
> element 
> (http://www.w3.org/html/wg/drafts/html/master/embedded-content-0.html#

> concept-media-load-resource)
>
>
says:
>
> "If either the media resource or the address of the current media 
> resource indicate a particular set of audio or video tracks to enable, 
> then the selected audio tracks must be enabled in the element's 
> audioTracks object, and, of the selected video tracks, the one that is 
> listed first in the element's videoTracks object must be selected."
>
> For the audio part it is pretty clear - all tracks that are "enabled" 
> should be used (and mixed) for playout. Video is less so.
> If there is more than one video track that is "enabled", which one 
> should be selected? There is no "first".
>
> Stefan
>
>
> On 3/21/13 7:46 AM, Adam Bergkvist wrote:
>> On 2013-03-20 18:12, Martin Thomson wrote:
>>> On 20 March 2013 07:33, Adam Bergkvist <adam.bergkvist@ericsson.com>
>>> wrote:
>>>> On 2013-03-19 14:00, Jim Barnett wrote:
>>>>>
>>>>> The final decision of what to display should be up to the sink.  
>>>>> If there are sinks that can handle MediaStreams containing 
>>>>> multiple audio or video tracks, then they should be able to use 
>>>>> whatever heuristics they want to select the
>>>>> Track(s) to play out.
>>>
>>> I'm concerned with the non-deterministic behaviour that could happen 
>>> if we don't make something explicit here.
>>
>> I agree. We need to do something about this since we removed the 
>> order of the tracks and there's no logical "first" track of each kind 
>> anymore.
>>
>> But if the sink has some mechanism for enumerating tracks and 
>> selecting which one to render, then that should just work for 
>> MediaStreams as well. However, selecting a track is no guarantee that 
>> there's data on it; it might just be muted on track-level anyhow.
>>
>> /Adam
>>
>
>

Received on Thursday, 21 March 2013 14:05:32 UTC