Re: Blobs and MediaStreams (Re: getUserMedia API: How to use the LocalMediaStream object ?)

On 03/07/2012 05:59 AM, Randell Jesup wrote:
> On 3/6/2012 6:21 PM, Robert O'Callahan wrote:
>> On Tue, Mar 6, 2012 at 3:44 PM, Randell Jesup <randell-ietf@jesup.org
>> <mailto:randell-ietf@jesup.org>> wrote:
>>
>>     So we need "something" to transform blobs into mediastreams (and
>>     the same principle would apply for recordings, at least of data
>>     from a camera/mic or decoded remote data).
>>
>>     There are a few ways to deal with this:
>>      1) self-identifying streams and a 'smart' decoder
>>      2) out-of-band information as to the encoding (i.e. specified by
>>     the app directly or indirectly)
>>
>>     Streams on the web typically are handled by out-of-band
>>     information (MIME types), so any solution we make should at least
>>     support that, and perhaps even use that as the primary solution.
>>
>>
>> A Blob object already has a MIME type along with the data. There is no
>> problem here.
>
> Good; I hadn't played much with Blobs and Harald's comment strongly
> implied (really stated) external metadata being required to decode it.
>
>>     I suppose (noodling without really knowing what I'm talking about
>>     ;-) ) you could have a disconnected <video> or <audio> element
>>     with a blob as a source with a mime type supplied, and add a
>>     'dest' or 'output' attribute which would be a MediaStream.  This
>>     might even make sense... And you'd have an obvious way to do
>>     things like provide initial scaling of the video, or insertion of
>>     subtitles, etc.  Hmmm.
>>
>>     I do want to get away from URL.createObjectURL(MediaStream) for
>>     setting the src for a <video> element, and I think there's a
>>     bigger opportunity here to unify media handling in a way that will
>>     make future work easier.
>>
>>
>> The MediaStreams Processing spec extends media elements' "src"
>> attribute so you can set it directly to a MediaStream: "video.src =
>> stream;". I've implemented that in my patches. It also adds an API to
>> capture a MediaStream from a media element, e.g. "stream =
>> video.captureStreamUntilEnded()".
>
> Right - I was forgetting the details of the Processing API proposal;
> that's what I was really suggesting here - we should use that portion of
> the MediaStream Processing API.  To quote myself in the message that
> kicked off this discussion:
>
>     URLs don't help us solve other problems (and add new ones);
>     MediaStreams are a mechanism that lets us solve a whole class of
>     problems (and encapsulate utility/semantics such as audio/video
>     synchronization and multiple/alternate  tracks).

My objections to the "video.src = stream" approach pretty much goes away 
if it would be possible to let the src-property be of type "any" (as 
Robert suggests in the MediaStream Processing API). I'm not really sure 
what the consequences are though.

It's pretty much the src attribute as a DOMString that I see issues 
with. E.g.

video1.src = stream;
log(video1.src); // "about:MediaStream"
video2.src = video1.src; // ??

>>
>> It would also make sense to extend the "src" attribute to directly
>> accept Blob objects as well. The behavior is just like assigning to
>> the result of createObjectURL, except it has better GC behavior since
>> you don't have to manually revoke the URL.
>>
>> Then you could do, e.g.,
>>   var video = document.createElement("video");
>>   video.src = blob;
>>   var stream = video.captureStreamUntilEnded();
>>   video.play();
>
> Rob's suggestion is exactly what I would like to push here, and similar
> to what I originally suggested - as mentioned above, this gives us a
> clean way to substitute data for cameras/mics when building
> MediaStreams; it promotes MediaStreams to being a common way to to
> handle audio and video (and maintain sync for them), and gives us a
> clean way to play back recorded blobs.  Also, if there's any need for
> the consumer of a MediaStream to affect the source, that could be fairly
> easily implemented with MediaStreams or things built on top of them;
> with URLs it may make it more painful.
>
> MediaStreams:
>      * Hold related audio and/or video tracks
>      * Can be created from Blobs via video elements (with Rob's
> suggestion above), cameras/mics (getUserMedia), or remote media streams
> (PeerConnections or from a remote URL via a video element with the
> MediaStream Processing API)

Is there a particular reason for involving a video element in this 
operation? An alternative could be:

var stream = new MediaStream(blob);

>      * Can be duplicated to feed content to multiple places
>      * Can be fed into media elements for playback (video.src =
> mediastream, which we're suggesting)
>      * Can be recorded into a Blob (mediastream.record())
>      * Can be encoded and sent (via PeerConnection)
>      * Can be processed in arbitrary manners (with the MediaStream
> Processing API)
>      * Allow for less-error-prone GCing of data by not needing to revoke
> URLs
>      * is a floor wax and a desert topping  ;-)

Topping comes in two flavors - vanilla and "Local". ;)

/Adam

Received on Wednesday, 7 March 2012 12:44:55 UTC