RE: MediaStreams and state changes

> -----Original Message-----
> From: Randell Jesup [mailto:randell-ietf@jesup.org]
> Sent: Thursday, June 7, 2012 7:18 AM
> 
> For discussion as part of the constraints issues today:
> 
> I have some unease about some parts of MediaStreams and getUserMedia
> that we haven't addressed or haven't discussed in detail.
> 
> One is changing video parameters after getUserMedia().

This is relevant to the other thread I started (http://lists.w3.org/Archives/Public/public-media-capture/2012Jul/0069.html)--sorry, somehow I missed this particular thread. I did, however, want to share some thought here as well as they seem relevant to the general conversation.

> Another is changing parameters of a stream that originates somewhere
> other than getUserMedia.  video elements using
> captureStreamUntilEnded(), PeerConnection-sourced streams (which might
> want to renegotiate based on the request, or send RTCP feedback, or use
> application-specific signaling), algorithmic generators (such as the Audio API),
> etc.
> 
> Right now, we've  somewhat waved our hands and said there'd be a method
> to re-do constraints after a MediaStream is created.  I generally am uneasy
> with this, and I'm especially uneasy when we're not talking about a
> getUserMedia()-originated stream.

So, now we have a proposal on the table for this. I took care to isolate the proposed API to a LocalMediaStream, and essentially punted on the option to allow downstream consumers to be able to apply change requests (directly through an API). I'd originally wanted to tie the API to the MediaStreamTrack, but then you allow any consumer (downstream or upstream) to be able to request changes, and these probably aren't relevant for Tracks that come from PeerConnection. What would be more ideal, is to have a derived LocalMediaStreamTrack object with these abilities, that is only created by getUserMedia, and whose interface cannot be obtained after forking the track (for example, such a local media stream track used in the constructor to a new MediaStream object would be handled only as a media stream track--losing its local-ness.)

 
> The code that originated the stream might be separated from the code
> consuming it that wants a change, so we need some way for a consumer of a
> MediaStream to indicate a need or request for changes in it.

I'm not so sure we actually do need a way for an arbitrary consumer to make a change. And if this need is necessary, then script libraries can write a request api should the contracts be so formal--unless there's something I'm missing here?

> Concerning re-doing the constraints call via a method on the mediastream
> (which had been vaguely proposed when we discussed this last):
>    a) re-doing the constraints has problems: the consumer of the stream may
> not be the caller to getUserMedia.  It might not even know who was.
>    b) implementing the full constraint algorithm and structure on all sources of
> MediaStreams is a considerable burden and might not match well

I agree with that. I don't think the generic MediaStream object is the right place for the API.

> If instead we pass a request back "upstream" (nominally as some form of
> event), it would let the event ripple up to the source (or sources) - this would
> apply at the track level generally.  There may be multiple MediaStream
> objects and processors between the consumer and the source.

This seems overly complex for the UA to support, and is something really simple that a script library could offer. As long as there is some way to do it, then the appropriate contracts can be worked out, even if the contract is as archaic as the human consumer asking the human producer (via the RTC video/audio channel itself) to turn up the volume on their end :)

> Also, fundamental object for this type of change is the track, and not the
> MediaStream, which needs to be thought about even if we were to use
> constraints for modifications.

Yep, which is why we should at least consider a "local" version of a MediaStreamTrack. The problem with my proposal (being on a LocalMediaStream object) is that the application of change requests through the proposed API is supposed to happen against the devices supplying the tracks, and is not necessarily tied to the actual tracks that are present in a LocalMediaStream at any given point in time (since Tracks lists are fully mutable). It's easy (and likely) that developers will confuse those concepts and think they are applying the requested changes to the current tracks. If we can limit this confusion it would be advantageous.

> My apologies for not having a concrete proposal; we don't have one on the
> table now either, and the vague comments make me worry we may leave
> this off too long (and the solution to it may impact the start-time constraints
> discussion).
> 
> --
> Randell Jesup
> randell-ietf@jesup.org

Received on Wednesday, 1 August 2012 22:54:59 UTC