RE: The Image Stream Processing pipeline/s

The existing way to do preview is the usual way of setting a  source of a <video> tag to be the MediaStreamTrack from which the image or frame is captured.  Something like

…
<p><video id="monitor" autoplay></video></p>
…
<script>
…
var vid = document.getElementById('monitor');
navigator.webkitGetUserMedia({video: true}, gotStream, noStream);
function gotStream(stream) {
            vid.src = URL.createObjectURL(stream);  // I assume this method will change based on the recent discussion regarding createObjectURL
            }
…
</script>

You can set constraints on the MediaStreamTrack, but these constraints are limited to what is listed in  http://dev.w3.org/2011/webrtc/editor/getusermedia.html#iana-registrations <%20http:/dev.w3.org/2011/webrtc/editor/getusermedia.html#iana-registrations> .  For instance, a camera preview stream could be at a lower FPS than one for a WebRTC video conferencing session.  {NOTE:  ‘zoom’ as a contraint does not seem to be resolved yet; I have filed a bug to get this issue definitively handled one way or another}

There was at one time a concept of a PictureDeviceTrack (see https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/proposals/SettingsAPI_proposal_v4.html#picturedevicetrack-interface ), which derived from a MediaStreamTrack, but the group did not carry this concept forward.  The current VideoStreamTrack (http://dev.w3.org/2011/webrtc/editor/getusermedia.html#videostreamtrack-interface) is the only track suitable for preview.

Probably not the answer you are looking for, but the group made a decision not to have photo-specific settings be a part of the VideoStreamTrack constraints set.

From: Shelly Lin [mailto:slin@mozilla.com]
Sent: Thursday, September 05, 2013 9:17 PM
To: Greg Billock
Cc: Rob Manson; public-media-capture@w3.org<mailto:public-media-capture@w3.org>
Subject: Re: The Image Stream Processing pipeline/s

Hi all,

A maybe related question. The draft of Image Capture says both getFrame() and takePhoto() capture a single still frame/image at one call, and the API also provides some options to manipulate the capture result. So should the Image Capture need to handle the preview of settings change in streaming mode? Like the camera preview window. Or this kind of streaming preview should be handled at user level, using setTimeout() or requestAnimationFrame()?

Thanks,
Shelly

________________________________
From: "Greg Billock" <gbillock@google.com<mailto:gbillock@google.com>>
To: "Rob Manson" <roBman@mob-labs.com<mailto:roBman@mob-labs.com>>
Cc: public-media-capture@w3.org<mailto:public-media-capture@w3.org>
Sent: Friday, September 6, 2013 12:12:20 AM
Subject: Re: The Image Stream Processing pipeline/s

I think compute-intensive image processing will not want to use the MediaRecorder pathway. It will likely not be presenting a frame-based chunk, and depending on what options are available you'd have to decode the stream. I think using something more like stream->video->canvas (although admittedly chewy) is better.


On Thu, Sep 5, 2013 at 8:51 AM, Rob Manson <roBman@mob-labs.com<mailto:roBman@mob-labs.com>> wrote:

So is the recommendation that we should use the Media Recording Blob based API?

  stream -> media recorder -> blob -> file reader -> array buffer

As compared to the current de-facto pipeline?

  stream -> <video> -> <canvas> -> image data -> array buffer

And Image Capture?  Giri has just published the latest version of that.


roBman



On 06/09/13 01:21, Harald Alvestrand wrote:

On 09/05/2013 07:37 AM, Rob Manson wrote:

Hi Adam,

that's true.  But Transferable objects avoid the copy overhead so
there is a real performance benefit in that.

Unless I'm missing something?
That's actually only true for browsers using shared memory between
entities.



roBman


On 05/09/13 15:14, Adam Bergkvist wrote:

On 2013-09-05 04:50, Rob Manson wrote:

Also since I don't believe Blob's are Transferable[9] then choosing B
has performance implications for processes that want to shift work off
into a Web Worker too (see comment above about B's secondary pipeline).

It's true that Blob's aren't Transferable (since they are immutable and
easy to share it doesn't add much), but you can send them as the first
(message) argument to postMessage() [1].

/Adam

[1] http://dev.w3.org/html5/postmsg/#dom-window-postmessage