W3C home > Mailing lists > Public > public-webrtc@w3.org > March 2012

Re: Blobs and MediaStreams (Re: getUserMedia API: How to use the LocalMediaStream object ?)

From: Adam Bergkvist <adam.bergkvist@ericsson.com>
Date: Wed, 7 Mar 2012 17:52:43 +0100
Message-ID: <4F57925B.3020804@ericsson.com>
To: Randell Jesup <randell-ietf@jesup.org>
CC: "public-webrtc@w3.org" <public-webrtc@w3.org>, Robert O'Callahan <roc@ocallahan.org>, "robert@ocallahan.org" <robert@ocallahan.org>
On 03/07/2012 04:19 PM, Randell Jesup wrote:
> On 3/7/2012 7:39 AM, Adam Bergkvist wrote:
>> On 03/07/2012 05:59 AM, Randell Jesup wrote:
>>> On 3/6/2012 6:21 PM, Robert O'Callahan wrote:
> [snip]
>>>> The MediaStreams Processing spec extends media elements' "src"
>>>> attribute so you can set it directly to a MediaStream: "video.src =
>>>> stream;". I've implemented that in my patches. It also adds an API to
>>>> capture a MediaStream from a media element, e.g. "stream =
>>>> video.captureStreamUntilEnded()".
> [snip]
>> My objections to the "video.src = stream" approach pretty much goes away
>> if it would be possible to let the src-property be of type "any" (as
>> Robert suggests in the MediaStream Processing API). I'm not really sure
>> what the consequences are though.
>>
>> It's pretty much the src attribute as a DOMString that I see issues
>> with. E.g.
>>
>> video1.src = stream;
>> log(video1.src); // "about:MediaStream"
>> video2.src = video1.src; // ??
>
> I'll let others chime in here.  How does this work if it's video2.src =
> stream;?Or with today's proposal:

I don't see any reasons why assigning stream to both video1.src and 
video2.src (as you mention above) wouldn't work. However, the example I 
provided in my previous mail was intended to highlight the behavior that 
you set a MediaStream, but you'll read back a DOMString. This changes to 
the better in Robert's proposal.

>     ms_url = URL.createObjectURL(stream);
>     video1.src = ms_url;
>     video2.src = ms_url;

The result of this depends if the URL was created with oneTimeOnly or 
not. oneTimeOnly is false by default so it will work. If set, the second 
assignment will fail since the URL would have been revoked after the 
first assignment.

> I think it's all under-specified currently and probably invalid; if so
> then I don't see it as a ding against the direct .src version - but I'm
> no expert here.
>
> One way that is specified and would work with the .src thing is to
> duplicate the stream, and then do video1.src = stream1; video2.src =
> stream2.  This would also work with createObjectURL().
>

True. That would give you the possibility to, e.g., enable and disable 
tracks independently between stream1 and stream2. If you don't need that 
extra control, you don't have to clone the stream.

>>>> It would also make sense to extend the "src" attribute to directly
>>>> accept Blob objects as well. The behavior is just like assigning to
>>>> the result of createObjectURL, except it has better GC behavior since
>>>> you don't have to manually revoke the URL.
>>>>
>>>> Then you could do, e.g.,
>>>> var video = document.createElement("video");
>>>> video.src = blob;
>>>> var stream = video.captureStreamUntilEnded();
>>>> video.play();
>>>
>>> Rob's suggestion is exactly what I would like to push here, and similar
>>> to what I originally suggested - as mentioned above, this gives us a
>>> clean way to substitute data for cameras/mics when building
>>> MediaStreams; it promotes MediaStreams to being a common way to to
>>> handle audio and video (and maintain sync for them), and gives us a
>>> clean way to play back recorded blobs. Also, if there's any need for
>>> the consumer of a MediaStream to affect the source, that could be fairly
>>> easily implemented with MediaStreams or things built on top of them;
>>> with URLs it may make it more painful.
>>>
>>> MediaStreams:
>>> * Hold related audio and/or video tracks
>>> * Can be created from Blobs via video elements (with Rob's
>>> suggestion above), cameras/mics (getUserMedia), or remote media streams
>>> (PeerConnections or from a remote URL via a video element with the
>>> MediaStream Processing API)
>>
>> Is there a particular reason for involving a video element in this
>> operation? An alternative could be:
>>
>> var stream = new MediaStream(blob);
>
> Well, that's would be doable, but by using the video element you do get
> all the things that come with that in terms of APIs (resizing,
> subtitles, etc) which might otherwise require duplicating a lot of API
> into MediaStream, but maybe this isn't a big deal.  I agree it doesn't
> have to be done via video elements, but that has the advantage of being
> able to capture a stream from a URL as well.
>

That may be the case, I think some use cases and examples would make it 
clearer.

> Overall, I think it's cleaner using video elements (and in the process
> gets us some other good things and straightforward usage).  Others may
> disagree.
>

It just felt like a bit strange to involve a media player in the 
conversion from a Blob to MediaStream. A bit like an old cassette deck 
player. Do I have to press play to start the conversion; can I use high 
speed dubbing? :)

>>> * Can be duplicated to feed content to multiple places
>>> * Can be fed into media elements for playback (video.src =
>>> mediastream, which we're suggesting)
>>> * Can be recorded into a Blob (mediastream.record())
>>> * Can be encoded and sent (via PeerConnection)
>>> * Can be processed in arbitrary manners (with the MediaStream
>>> Processing API)
>>> * Allow for less-error-prone GCing of data by not needing to revoke
>>> URLs
>>> * is a floor wax and a desert topping ;-)
>>
>> Topping comes in two flavors - vanilla and "Local". ;)
>
> I prefer chopped nuts on top, which requires Processing ;-)
>
Received on Wednesday, 7 March 2012 16:58:32 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 7 March 2012 16:58:33 GMT