Re: Blobs and MediaStreams (Re: getUserMedia API: How to use the LocalMediaStream object ?)

On 3/6/2012 6:21 PM, Robert O'Callahan wrote:
> On Tue, Mar 6, 2012 at 3:44 PM, Randell Jesup <randell-ietf@jesup.org 
> <mailto:randell-ietf@jesup.org>> wrote:
>
>     So we need "something" to transform blobs into mediastreams (and
>     the same principle would apply for recordings, at least of data
>     from a camera/mic or decoded remote data).
>
>     There are a few ways to deal with this:
>      1) self-identifying streams and a 'smart' decoder
>      2) out-of-band information as to the encoding (i.e. specified by
>     the app directly or indirectly)
>
>     Streams on the web typically are handled by out-of-band
>     information (MIME types), so any solution we make should at least
>     support that, and perhaps even use that as the primary solution.
>
>
> A Blob object already has a MIME type along with the data. There is no 
> problem here.

Good; I hadn't played much with Blobs and Harald's comment strongly 
implied (really stated) external metadata being required to decode it.

>     I suppose (noodling without really knowing what I'm talking about
>     ;-) ) you could have a disconnected <video> or <audio> element
>     with a blob as a source with a mime type supplied, and add a
>     'dest' or 'output' attribute which would be a MediaStream.  This
>     might even make sense... And you'd have an obvious way to do
>     things like provide initial scaling of the video, or insertion of
>     subtitles, etc.  Hmmm.
>
>     I do want to get away from URL.createObjectURL(MediaStream) for
>     setting the src for a <video> element, and I think there's a
>     bigger opportunity here to unify media handling in a way that will
>     make future work easier.
>
>
> The MediaStreams Processing spec extends media elements' "src" 
> attribute so you can set it directly to a MediaStream: "video.src = 
> stream;". I've implemented that in my patches. It also adds an API to 
> capture a MediaStream from a media element, e.g. "stream = 
> video.captureStreamUntilEnded()".

Right - I was forgetting the details of the Processing API proposal; 
that's what I was really suggesting here - we should use that portion of 
the MediaStream Processing API.  To quote myself in the message that 
kicked off this discussion:

    URLs don't help us solve other problems (and add new ones);
    MediaStreams are a mechanism that lets us solve a whole class of
    problems (and encapsulate utility/semantics such as audio/video
    synchronization and multiple/alternate  tracks).


>
> It would also make sense to extend the "src" attribute to directly 
> accept Blob objects as well. The behavior is just like assigning to 
> the result of createObjectURL, except it has better GC behavior since 
> you don't have to manually revoke the URL.
>
> Then you could do, e.g.,
>   var video = document.createElement("video");
>   video.src = blob;
>   var stream = video.captureStreamUntilEnded();
>   video.play();

Rob's suggestion is exactly what I would like to push here, and similar 
to what I originally suggested - as mentioned above, this gives us a 
clean way to substitute data for cameras/mics when building 
MediaStreams; it promotes MediaStreams to being a common way to to 
handle audio and video (and maintain sync for them), and gives us a 
clean way to play back recorded blobs.  Also, if there's any need for 
the consumer of a MediaStream to affect the source, that could be fairly 
easily implemented with MediaStreams or things built on top of them; 
with URLs it may make it more painful.

MediaStreams:
     * Hold related audio and/or video tracks
     * Can be created from Blobs via video elements (with Rob's 
suggestion above), cameras/mics (getUserMedia), or remote media streams 
(PeerConnections or from a remote URL via a video element with the 
MediaStream Processing API)
     * Can be duplicated to feed content to multiple places
     * Can be fed into media elements for playback (video.src = 
mediastream, which we're suggesting)
     * Can be recorded into a Blob (mediastream.record())
     * Can be encoded and sent (via PeerConnection)
     * Can be processed in arbitrary manners (with the MediaStream 
Processing API)
     * Allow for less-error-prone GCing of data by not needing to revoke 
URLs
     * is a floor wax and a desert topping  ;-)

-- 
Randell Jesup
randell-ietf@jesup.org