Re: Advice on extending CanvasRenderingContext2D to support Depth Streams

Hi Justin,

> Just one questionthough:
> do you really need to use a  canvas at all? It seems like an
 > unnecessary indirection to me. For the use case in that example code
 > snippet, I thinkt what you would really want is an API for extracting
 > the DepthData of the current frame directly from the video element?

I totally agree and this doesn't just go for DepthData but for all video 
frame pixels. Personally, I'd really like to be able to do something like:

var uint16pixels = video.getDepthTracks()[0].getPixels({ 
bitsPerPixel:16, channelsPerPixel:1 });

AND

var uint8pixels = video.getVideoTracks()[0].getPixels({ bitsPerPixel:8, 
channelsPerPixel:4 });

However, canvas is really the only way to currently access video frame 
pixels so we have been focused on that.

I'd be interested to hear people's general thoughts on something like 
I've described above - even though I know that's not directly related to 
canvas, I'm sure there's a lot of relevant experience on this list 8)


> Also, you stated that the  example should probably use rAF instead of
 > setInterval. I would suggest that depth data capture loop should be
 > tied to the incoming frame of the video stream, not to the display
 > frame rate. perhaps you should be using a listener attached to the
 > video element's 'timeupdate' event.

OK now you've really got me started 8)

I totally agree and it seems wildly inefficient to be processing such 
compute intensive pixel extraction on what might be duplicates (e.g. 
processing the same frame multiple times).

Ideally we'd set an ontimeupdate event handler as you suggest and that 
would then check to make sure readyState > 1 and we should be golden. 
But in reality this isn't the case and the ontimeupdate event is really 
only thrown about 3-4 times per second. This is obviously well below the 
24-30fps of most common video.

Or this could even turn my .getPixels() sketch above into a onnewframe 
event that would just automatically be passed the pixels from the newly 
minted video frame.

But for now we are left using rAF and then comparing currentTime and 
readyState to try to minimse the amount of re-processing we're doing.

There's also the developing Image Capture API[1] that is designed to 
include a grabFrame() method. But you'll see the example in that spec 
uses setInterval and the strong feedback from the other members of the 
Working Group/Task Force when I raised this a year or two ago was that 
setInterval/rAF are the way this should be handled.

I'd love to hear support to the contrary from "anyone" else 8)

Sorry if this seems like a bit of a rant but it's a really key issue for 
me that extends beyond the Depth Camera work and into all Computer 
Vision on the Web Platform.

roBman

[1] http://w3c.github.io/mediacapture-image/index.html

Received on Thursday, 13 November 2014 00:50:13 UTC