Re: The Image Stream Processing pipeline/s

I think it's helpful to think of MediaRecorder not as a recorder per se,
but as a way to encode/mux a number of streams to a specific format. It's
more a MediaTranscoder :)

The piece that's missing for your "new" pipeline, i.e. one that doesn't
need to route through <canvas> and <video> elements is a way to go from
MediaStream to a continuous sequence of e.g. ImageData objects, and back
for the final encoding. I'm tempted to call those "WebVideoNode" objects,
since this is an idea very similar to WebAudio processing.

I'm not aware anybody is working on this right now, but it certainly would
be a very useful thing to have.

 - rachel



On Thu, Sep 5, 2013 at 9:12 AM, Greg Billock <gbillock@google.com> wrote:

> I think compute-intensive image processing will not want to use the
> MediaRecorder pathway. It will likely not be presenting a frame-based
> chunk, and depending on what options are available you'd have to decode the
> stream. I think using something more like stream->video->canvas (although
> admittedly chewy) is better.
>
>
> On Thu, Sep 5, 2013 at 8:51 AM, Rob Manson <roBman@mob-labs.com> wrote:
>
>> So is the recommendation that we should use the Media Recording Blob
>> based API?
>>
>>   stream -> media recorder -> blob -> file reader -> array buffer
>>
>> As compared to the current de-facto pipeline?
>>
>>   stream -> <video> -> <canvas> -> image data -> array buffer
>>
>> And Image Capture?  Giri has just published the latest version of that.
>>
>>
>> roBman
>>
>>
>>
>> On 06/09/13 01:21, Harald Alvestrand wrote:
>>
>>> On 09/05/2013 07:37 AM, Rob Manson wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> that's true.  But Transferable objects avoid the copy overhead so
>>>> there is a real performance benefit in that.
>>>>
>>>> Unless I'm missing something?
>>>>
>>> That's actually only true for browsers using shared memory between
>>> entities.
>>>
>>>
>>>> roBman
>>>>
>>>>
>>>> On 05/09/13 15:14, Adam Bergkvist wrote:
>>>>
>>>>> On 2013-09-05 04:50, Rob Manson wrote:
>>>>>
>>>>>> Also since I don't believe Blob's are Transferable[9] then choosing B
>>>>>> has performance implications for processes that want to shift work off
>>>>>> into a Web Worker too (see comment above about B's secondary
>>>>>> pipeline).
>>>>>>
>>>>>
>>>>> It's true that Blob's aren't Transferable (since they are immutable and
>>>>> easy to share it doesn't add much), but you can send them as the first
>>>>> (message) argument to postMessage() [1].
>>>>>
>>>>> /Adam
>>>>>
>>>>> [1] http://dev.w3.org/html5/**postmsg/#dom-window-**postmessage<http://dev.w3.org/html5/postmsg/#dom-window-postmessage>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>