Re: MediaStreams and state changes

On 8/1/2012 3:54 PM, Travis Leithead wrote:
>> -----Original Message-----
>> From: Randell Jesup [mailto:randell-ietf@jesup.org]
>> Sent: Thursday, June 7, 2012 7:18 AM
>>
>> For discussion as part of the constraints issues today:
>>
>> I have some unease about some parts of MediaStreams and getUserMedia
>> that we haven't addressed or haven't discussed in detail.
>>
>> One is changing video parameters after getUserMedia().
> This is relevant to the other thread I started (http://lists.w3.org/Archives/Public/public-media-capture/2012Jul/0069.html)--sorry, somehow I missed this particular thread. I did, however, want to share some thought here as well as they seem relevant to the general conversation.

I did also raise this in one of the Telco's, but I'm not sure you were 
on that one.

>> Another is changing parameters of a stream that originates somewhere
>> other than getUserMedia.  video elements using
>> captureStreamUntilEnded(), PeerConnection-sourced streams (which might
>> want to renegotiate based on the request, or send RTCP feedback, or use
>> application-specific signaling), algorithmic generators (such as the Audio API),
>> etc.
>>
>> Right now, we've  somewhat waved our hands and said there'd be a method
>> to re-do constraints after a MediaStream is created.  I generally am uneasy
>> with this, and I'm especially uneasy when we're not talking about a
>> getUserMedia()-originated stream.
> So, now we have a proposal on the table for this. I took care to isolate the proposed API to a LocalMediaStream, and essentially punted on the option to allow downstream consumers to be able to apply change requests (directly through an API). I'd originally wanted to tie the API to the MediaStreamTrack, but then you allow any consumer (downstream or upstream) to be able to request changes, and these probably aren't relevant for Tracks that come from PeerConnection.

These sorts of requests are frequently applicable to PeerConnections.  
The simplest version of such a request, a request to change resolution, 
would make total sense for PeerConnection-sourced streams - in fact 
we're trying to standardize the wire protocol for doing this (currently 
it appears there's consensus for doing it in SDP).  Others might not be 
handled by the system directly, but an application could catch the 
requests and broker them to the other party over DataChannels (or 
signaling). For example, think of a WebRTC-enabled security system where 
you could switch cameras, tell cameras to change parameters, etc.

Anything that isn't relevant can and would simply be ignored.

> What would be more ideal, is to have a derived LocalMediaStreamTrack object with these abilities, that is only created by getUserMedia, and whose interface cannot be obtained after forking the track (for example, such a local media stream track used in the constructor to a new MediaStream object would be handled only as a media stream track--losing its local-ness.)

This may well work as well; it would be interesting to compare the 
differences produced or implied by these two different APIs.

>> The code that originated the stream might be separated from the code
>> consuming it that wants a change, so we need some way for a consumer of a
>> MediaStream to indicate a need or request for changes in it.
> I'm not so sure we actually do need a way for an arbitrary consumer to make a change. And if this need is necessary, then script libraries can write a request api should the contracts be so formal--unless there's something I'm missing here?

You may not be missing anything; I'm no expert on script libraries and 
what's capable.  My concern with reusing the constraint API was that if 
you are using components of different origin to build an app, the code 
consuming the MediaStream may be significantly isolated from the code 
that set up the capture.  My secondary concern is that you'd need to 
implement code for taking a request/need and editing an existing 
constraint structure to include this new request - a non-trivial 
operation I suspect.

I also dislike taking problems and dumping them into the responsibility 
of some future script library...

>> Concerning re-doing the constraints call via a method on the mediastream
>> (which had been vaguely proposed when we discussed this last):
>>     a) re-doing the constraints has problems: the consumer of the stream may
>> not be the caller to getUserMedia.  It might not even know who was.
>>     b) implementing the full constraint algorithm and structure on all sources of
>> MediaStreams is a considerable burden and might not match well
> I agree with that. I don't think the generic MediaStream object is the right place for the API.
>
>> If instead we pass a request back "upstream" (nominally as some form of
>> event), it would let the event ripple up to the source (or sources) - this would
>> apply at the track level generally.  There may be multiple MediaStream
>> objects and processors between the consumer and the source.
> This seems overly complex for the UA to support, and is something really simple that a script library could offer.

Perhaps I miss something, but this is confusing: overly complex for the 
UA, but really simple for a script library.

>   As long as there is some way to do it, then the appropriate contracts can be worked out, even if the contract is as archaic as the human consumer asking the human producer (via the RTC video/audio channel itself) to turn up the volume on their end :)
>
>> Also, fundamental object for this type of change is the track, and not the
>> MediaStream, which needs to be thought about even if we were to use
>> constraints for modifications.
> Yep, which is why we should at least consider a "local" version of a MediaStreamTrack. The problem with my proposal (being on a LocalMediaStream object) is that the application of change requests through the proposed API is supposed to happen against the devices supplying the tracks, and is not necessarily tied to the actual tracks that are present in a LocalMediaStream at any given point in time (since Tracks lists are fully mutable). It's easy (and likely) that developers will confuse those concepts and think they are applying the requested changes to the current tracks. If we can limit this confusion it would be advantageous.

Yes, I worry about the confusion as well.  A clear and easy-to-use API 
for applications to vary capture parameters seems important IMHO, 
whatever it is.

-- 
Randell Jesup
randell-ietf@jesup.org

Received on Thursday, 2 August 2012 08:07:44 UTC