Re: The Image Stream Processing pipeline/s

Hrm...Web Video as a peer to Web Audio...that's interesting 8)

A WebVideoNode (e.g. Web Video equivalent of the Web Audio 
ScriptProcessorNode) is a very interesting idea I'd be really happy to 
collaborate on 8)

I think allowing developers to use the same mental model across both 
Audio and Video post-processing would be a real step forward.

Having a general model of ScriptProcessorNodes that can sit on top of 
streams through specified Array Buffers is just plain useful.  The same 
model could then be used for processing sensor and meta data streams in 
Array Buffers that come in through RTC Data Channels, Web Sockets and 
XHR...or even the File API too.

roBman


On 07/09/13 03:43, Rachel Blum wrote:
> I think it's helpful to think of MediaRecorder not as a recorder per se,
> but as a way to encode/mux a number of streams to a specific format.
> It's more a MediaTranscoder :)
>
> The piece that's missing for your "new" pipeline, i.e. one that doesn't
> need to route through <canvas> and <video> elements is a way to go from
> MediaStream to a continuous sequence of e.g. ImageData objects, and back
> for the final encoding. I'm tempted to call those "WebVideoNode"
> objects, since this is an idea very similar to WebAudio processing.
>
> I'm not aware anybody is working on this right now, but it certainly
> would be a very useful thing to have.
>
>   - rachel
>
>
>
> On Thu, Sep 5, 2013 at 9:12 AM, Greg Billock <gbillock@google.com
> <mailto:gbillock@google.com>> wrote:
>
>     I think compute-intensive image processing will not want to use the
>     MediaRecorder pathway. It will likely not be presenting a
>     frame-based chunk, and depending on what options are available you'd
>     have to decode the stream. I think using something more like
>     stream->video->canvas (although admittedly chewy) is better.
>
>
>     On Thu, Sep 5, 2013 at 8:51 AM, Rob Manson <roBman@mob-labs.com
>     <mailto:roBman@mob-labs.com>> wrote:
>
>         So is the recommendation that we should use the Media Recording
>         Blob based API?
>
>            stream -> media recorder -> blob -> file reader -> array buffer
>
>         As compared to the current de-facto pipeline?
>
>            stream -> <video> -> <canvas> -> image data -> array buffer
>
>         And Image Capture?  Giri has just published the latest version
>         of that.
>
>
>         roBman
>
>
>
>         On 06/09/13 01:21, Harald Alvestrand wrote:
>
>             On 09/05/2013 07:37 AM, Rob Manson wrote:
>
>                 Hi Adam,
>
>                 that's true.  But Transferable objects avoid the copy
>                 overhead so
>                 there is a real performance benefit in that.
>
>                 Unless I'm missing something?
>
>             That's actually only true for browsers using shared memory
>             between
>             entities.
>
>
>                 roBman
>
>
>                 On 05/09/13 15:14, Adam Bergkvist wrote:
>
>                     On 2013-09-05 04:50, Rob Manson wrote:
>
>                         Also since I don't believe Blob's are
>                         Transferable[9] then choosing B
>                         has performance implications for processes that
>                         want to shift work off
>                         into a Web Worker too (see comment above about
>                         B's secondary pipeline).
>
>
>                     It's true that Blob's aren't Transferable (since
>                     they are immutable and
>                     easy to share it doesn't add much), but you can send
>                     them as the first
>                     (message) argument to postMessage() [1].
>
>                     /Adam
>
>                     [1]
>                     http://dev.w3.org/html5/__postmsg/#dom-window-__postmessage
>                     <http://dev.w3.org/html5/postmsg/#dom-window-postmessage>
>
>
>
>
>
>
>
>

Received on Saturday, 7 September 2013 01:02:48 UTC