Re: Async Image -> ImageData conversion

I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:

https://www.scirra.com/labs/specs/imagedata-blob-extensions.html

Let me know if you have any feedback on this.



On 18 June 2015 at 22:27, Travis Leithead <travis.leithead@microsoft.com>
wrote:

>  Cool. Want to write it up as a small extension spec? We have a new
> Incubator Community Group coming up soon that would be an ideal place to
> drop a small spec into and let implementers experiment with it and get
> broader community feedback on the idea.
>
>
>
> *From:* ash@scirra.com [mailto:ash@scirra.com] *On Behalf Of *Ashley
> Gullen
> *Sent:* Wednesday, June 17, 2015 2:06 PM
> *To:* Travis Leithead
> *Cc:* public-webapps@w3.org
> *Subject:* Re: Async Image -> ImageData conversion
>
>
>
> That seems like a good start. I suppose there should be a
> putImageDataAsync counterpart too? Then we can do:
>
>
>
> Blob -> Image via: load blob URL
>
> Blob -> ImageData via: load blob URL -> Canvas drawImage
> -> getImageDataAsync
>
> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>
> Image -> ImageData via: Canvas drawImage -> getImageDataAsync
>
> ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
>
> ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load
> blob URL
>
>
>
> I think the potential problems with this are:
>
> - putImageDataAsync is the logical counterpart to getImageDataAsync, but
> what happens if you make other synchronous draw calls while
> putImageDataAsync is processing? What actually ends up in the canvas and
> how is this defined?
>
> - some of the conversion steps seem pretty long winded, in particular Blob
> -> ImageData and ImageData -> Image. If implementors think they can
> optimise these to be as efficient as direct conversion that's fine, but is
> that actually doable with such a roundabout API?
>
>
>
> If ImageData has:
>
> Promise<Blob> toBlob();
>
> Promise<ImageData> fromBlob(blob);
>
>
>
> then we can use the following conversions without new canvas functions:
>
>
>
> Blob -> Image via: load blob URL
>
> Blob -> ImageData via: ImageData.fromBlob
>
> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>
> Image -> ImageData via: Canvas drawImage -> Canvas toBlob ->
> ImageData.fromBlob
>
> ImageData -> Blob via: ImageData.toBlob
>
> ImageData -> Image via: ImageData.toBlob -> load blob URL
>
>
>
> That looks like a much more reasonable set of conversions to me, and is
> all async.
>
>
>
> I was thinking about toImage and fromImage, but it seems to be a bigger
> step to add a new way to make images, and there's the question of what the
> src for the image returned by toImage should be.
>
>
>
>
>
>
>
>
>
> On 17 June 2015 at 18:59, Travis Leithead <travis.leithead@microsoft.com>
> wrote:
>
>  I think solving at least the first-order problem of extracting data from
> the Canvas async is do-able.
>
>
>
> Something like:
>
>
>
> Promise<ImageData> getImageDataAsync(x,y,w,h);
>
>
>
> seems sensible to add J
>
>
>
> *From:* ash@scirra.com [mailto:ash@scirra.com] *On Behalf Of *Ashley
> Gullen
> *Sent:* Wednesday, June 17, 2015 10:00 AM
> *To:* public-webapps@w3.org
> *Subject:* Async Image -> ImageData conversion
>
>
>
> I was wondering if there is anything on the standards track to
> asynchronously get an ImageData object from an Image?
>
>
>
> We have a web app dealing with large sprite sheets (up to 2048x2048), and
> at some point we need to get an ImageData for this. Currently the only way
> I know of doing this is via a canvas 2d context like so:
>
>
>
> function ImageToImageData(img)
>
> {
>
>             let w = img.width;
>
>             let h = img.height;
>
>
>
>             let canvas = document.createElement("canvas");
>
>             canvas.width = w;
>
>             canvas.height = h;
>
>             let ctx = canvas.getContext("2d");
>
>             ctx.drawImage(img, 0, 0);
>
>             return ctx.getImageData(0, 0, w, h);
>
> };
>
>
>
> This whole function is synchronous. With large images, in Chrome the
> getImageData call can jank for well over 100ms on a high-end desktop
> machine. Maybe this can be faster but I really doubt this whole function is
> likely to be done in <16ms on low-end devices, so I believe there ought to
> be an async alternative. As far as I am aware there is none.
>
>
>
> Ideally we could have an async ImageData.fromImage(img) method or similar.
> It would also be useful if we could decode directly from Blob to ImageData
> without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
> ?).
>
>
>
> More broadly, it would be good to have a comprehensive set of async
> conversion functions between Blob, Image, and ImageData. Blob -> Image can
> already be done by setting the image src to a blob URL, but basically
> everything else requires going synchronously through a 2D canvas, which
> janks apps which have to process lots of large images. Running a canvas in
> a worker could possibly help, but that would require both Blob and Image to
> be transferable to be useful.
>
>
>
> Ashley Gullen
>
> Scirra.com
>
>
>
>
>