RE: Async Image -> ImageData conversion

Cool. Want to write it up as a small extension spec? We have a new Incubator Community Group coming up soon that would be an ideal place to drop a small spec into and let implementers experiment with it and get broader community feedback on the idea.

From: ash@scirra.com [mailto:ash@scirra.com] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 2:06 PM
To: Travis Leithead
Cc: public-webapps@w3.org
Subject: Re: Async Image -> ImageData conversion

That seems like a good start. I suppose there should be a putImageDataAsync counterpart too? Then we can do:

Blob -> Image via: load blob URL
Blob -> ImageData via: load blob URL -> Canvas drawImage -> getImageDataAsync
Image -> Blob via: Canvas drawImage -> Canvas toBlob
Image -> ImageData via: Canvas drawImage -> getImageDataAsync
ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load blob URL

I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but what happens if you make other synchronous draw calls while putImageDataAsync is processing? What actually ends up in the canvas and how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob -> ImageData and ImageData -> Image. If implementors think they can optimise these to be as efficient as direct conversion that's fine, but is that actually doable with such a roundabout API?

If ImageData has:
Promise<Blob> toBlob();
Promise<ImageData> fromBlob(blob);

then we can use the following conversions without new canvas functions:

Blob -> Image via: load blob URL
Blob -> ImageData via: ImageData.fromBlob
Image -> Blob via: Canvas drawImage -> Canvas toBlob
Image -> ImageData via: Canvas drawImage -> Canvas toBlob -> ImageData.fromBlob
ImageData -> Blob via: ImageData.toBlob
ImageData -> Image via: ImageData.toBlob -> load blob URL

That looks like a much more reasonable set of conversions to me, and is all async.

I was thinking about toImage and fromImage, but it seems to be a bigger step to add a new way to make images, and there's the question of what the src for the image returned by toImage should be.




On 17 June 2015 at 18:59, Travis Leithead <travis.leithead@microsoft.com<mailto:travis.leithead@microsoft.com>> wrote:
I think solving at least the first-order problem of extracting data from the Canvas async is do-able.

Something like:

Promise<ImageData> getImageDataAsync(x,y,w,h);

seems sensible to add ☺

From: ash@scirra.com<mailto:ash@scirra.com> [mailto:ash@scirra.com<mailto:ash@scirra.com>] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 10:00 AM
To: public-webapps@w3.org<mailto:public-webapps@w3.org>
Subject: Async Image -> ImageData conversion

I was wondering if there is anything on the standards track to asynchronously get an ImageData object from an Image?

We have a web app dealing with large sprite sheets (up to 2048x2048), and at some point we need to get an ImageData for this. Currently the only way I know of doing this is via a canvas 2d context like so:

function ImageToImageData(img)
{
            let w = img.width;
            let h = img.height;

            let canvas = document.createElement("canvas");
            canvas.width = w;
            canvas.height = h;
            let ctx = canvas.getContext("2d");
            ctx.drawImage(img, 0, 0);
            return ctx.getImageData(0, 0, w, h);
};

This whole function is synchronous. With large images, in Chrome the getImageData call can jank for well over 100ms on a high-end desktop machine. Maybe this can be faster but I really doubt this whole function is likely to be done in <16ms on low-end devices, so I believe there ought to be an async alternative. As far as I am aware there is none.

Ideally we could have an async ImageData.fromImage(img) method or similar. It would also be useful if we could decode directly from Blob to ImageData without having to go via an Image as well (perhaps ImageData.fromBlob(blob) ?).

More broadly, it would be good to have a comprehensive set of async conversion functions between Blob, Image, and ImageData. Blob -> Image can already be done by setting the image src to a blob URL, but basically everything else requires going synchronously through a 2D canvas, which janks apps which have to process lots of large images. Running a canvas in a worker could possibly help, but that would require both Blob and Image to be transferable to be useful.

Ashley Gullen
Scirra.com