RE: [mediacapture-image] Need to consider GPU copy (#8)

Since this request is dealing with a broader topic than the typical issue posted onto the GH repo’s, I am moving discussion away from GH and onto the Media Capture mailing list.  Therefore I am replicating the original issue from GH at the end of this email.

Thanks Dongseong for your helpful comments.


> Currently Mediastream Image Capture spec forces gpu based video to read back to system memory. It hits performance very badly.

> First of all, FrameGrabEvent event inherently return software RGBA memory block. It should return handle to point out video frame.
The motivation behind returning an ImageData object was to allow developers to manipulate RGBA data directly (some use cases were discussed in https://lists.w3.org/Archives/Public/public-media-capture/2013Feb/0067.html).  The example in the spec does not show this kind of processing in Javascript, so I can see how an independent reader would come to the conclusion that allocating browser heap to a block of memory for ImageData would be wasteful.

Nevertheless, that is not the intended purpose of grabFrame.  I also don’t see how your proposal would allow JS to get access to the RGBA data corresponding to the captured image, while still avoiding the system memory hit that you are concerned with.  Can you expand?

-Giri Mandyam, ImageCapture spec editor

From: Dongseong Hwang [mailto:notifications@github.com]
Sent: Wednesday, March 04, 2015 7:28 AM
To: w3c/mediacapture-image
Subject: [mediacapture-image] Need to consider GPU copy (#8)


Currently Mediastream Image Capture spec forces gpu based video to read back to system memory. It hits performance very badly.

First of all, FrameGrabEvent event inherently return software RGBA memory block. It should return handle to point out video frame.

interface FrameGrabEvent : Event {

    readonly    attribute ImageData imageData;

};

In following example, canvas.getContext('2d').drawImage(imgData, ...) is way inefficient than canvas.getContext('2d').drawImage(videoElement, ...).
If video and canvas is gpu accelerated, canvas.getContext('2d').drawImage(videoElement, ...) can copy gpu texture to gpu texture. but canvas.getContext('2d').drawImage(imgData, ...) requires to read back gpu texture to cpu memory and then upload cpu memory to gpu texture.

   if (captureDevice) {

         frameVar = setInterval(captureDevice.grabFrame().then(processFrame()), 1000);

         }

     }



 function processFrame(e) {

     imgData = e.imageData;

     canvas.width = imgData.width;

     canvas.height = imgData.height;

     canvas.getContext('2d').drawImage(imgData, 0, 0,imgData.width,imgData.height);

     }

in the same sense, it's very inefficient to upload the captured frame to WebGL or WebCL.

—
Reply to this email directly or view it on GitHub<https://github.com/w3c/mediacapture-image/issues/8>.