Re: Async Image -> ImageData conversion

I was advised to mark this as an "Editor's draft", and have done so. I also
added a new proposed ImageData.fromImage method, with rationale included in
the document. It's at the same URL:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html


On 19 June 2015 at 13:43, Ashley Gullen <ashley@scirra.com> wrote:

> I've not done this before, so I've no idea if this is the right/useful
> approach, but I drafted a spec for it here:
>
> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>
> Let me know if you have any feedback on this.
>
>
>
> On 18 June 2015 at 22:27, Travis Leithead <travis.leithead@microsoft.com>
> wrote:
>
>>  Cool. Want to write it up as a small extension spec? We have a new
>> Incubator Community Group coming up soon that would be an ideal place to
>> drop a small spec into and let implementers experiment with it and get
>> broader community feedback on the idea.
>>
>>
>>
>> *From:* ash@scirra.com [mailto:ash@scirra.com] *On Behalf Of *Ashley
>> Gullen
>> *Sent:* Wednesday, June 17, 2015 2:06 PM
>> *To:* Travis Leithead
>> *Cc:* public-webapps@w3.org
>> *Subject:* Re: Async Image -> ImageData conversion
>>
>>
>>
>> That seems like a good start. I suppose there should be a
>> putImageDataAsync counterpart too? Then we can do:
>>
>>
>>
>> Blob -> Image via: load blob URL
>>
>> Blob -> ImageData via: load blob URL -> Canvas drawImage
>> -> getImageDataAsync
>>
>> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>>
>> Image -> ImageData via: Canvas drawImage -> getImageDataAsync
>>
>> ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
>>
>> ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load
>> blob URL
>>
>>
>>
>> I think the potential problems with this are:
>>
>> - putImageDataAsync is the logical counterpart to getImageDataAsync, but
>> what happens if you make other synchronous draw calls while
>> putImageDataAsync is processing? What actually ends up in the canvas and
>> how is this defined?
>>
>> - some of the conversion steps seem pretty long winded, in particular
>> Blob -> ImageData and ImageData -> Image. If implementors think they can
>> optimise these to be as efficient as direct conversion that's fine, but is
>> that actually doable with such a roundabout API?
>>
>>
>>
>> If ImageData has:
>>
>> Promise<Blob> toBlob();
>>
>> Promise<ImageData> fromBlob(blob);
>>
>>
>>
>> then we can use the following conversions without new canvas functions:
>>
>>
>>
>> Blob -> Image via: load blob URL
>>
>> Blob -> ImageData via: ImageData.fromBlob
>>
>> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>>
>> Image -> ImageData via: Canvas drawImage -> Canvas toBlob ->
>> ImageData.fromBlob
>>
>> ImageData -> Blob via: ImageData.toBlob
>>
>> ImageData -> Image via: ImageData.toBlob -> load blob URL
>>
>>
>>
>> That looks like a much more reasonable set of conversions to me, and is
>> all async.
>>
>>
>>
>> I was thinking about toImage and fromImage, but it seems to be a bigger
>> step to add a new way to make images, and there's the question of what the
>> src for the image returned by toImage should be.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 17 June 2015 at 18:59, Travis Leithead <travis.leithead@microsoft.com>
>> wrote:
>>
>>  I think solving at least the first-order problem of extracting data
>> from the Canvas async is do-able.
>>
>>
>>
>> Something like:
>>
>>
>>
>> Promise<ImageData> getImageDataAsync(x,y,w,h);
>>
>>
>>
>> seems sensible to add J
>>
>>
>>
>> *From:* ash@scirra.com [mailto:ash@scirra.com] *On Behalf Of *Ashley
>> Gullen
>> *Sent:* Wednesday, June 17, 2015 10:00 AM
>> *To:* public-webapps@w3.org
>> *Subject:* Async Image -> ImageData conversion
>>
>>
>>
>> I was wondering if there is anything on the standards track to
>> asynchronously get an ImageData object from an Image?
>>
>>
>>
>> We have a web app dealing with large sprite sheets (up to 2048x2048), and
>> at some point we need to get an ImageData for this. Currently the only way
>> I know of doing this is via a canvas 2d context like so:
>>
>>
>>
>> function ImageToImageData(img)
>>
>> {
>>
>>             let w = img.width;
>>
>>             let h = img.height;
>>
>>
>>
>>             let canvas = document.createElement("canvas");
>>
>>             canvas.width = w;
>>
>>             canvas.height = h;
>>
>>             let ctx = canvas.getContext("2d");
>>
>>             ctx.drawImage(img, 0, 0);
>>
>>             return ctx.getImageData(0, 0, w, h);
>>
>> };
>>
>>
>>
>> This whole function is synchronous. With large images, in Chrome the
>> getImageData call can jank for well over 100ms on a high-end desktop
>> machine. Maybe this can be faster but I really doubt this whole function is
>> likely to be done in <16ms on low-end devices, so I believe there ought to
>> be an async alternative. As far as I am aware there is none.
>>
>>
>>
>> Ideally we could have an async ImageData.fromImage(img) method or
>> similar. It would also be useful if we could decode directly from Blob to
>> ImageData without having to go via an Image as well (perhaps
>> ImageData.fromBlob(blob) ?).
>>
>>
>>
>> More broadly, it would be good to have a comprehensive set of async
>> conversion functions between Blob, Image, and ImageData. Blob -> Image can
>> already be done by setting the image src to a blob URL, but basically
>> everything else requires going synchronously through a 2D canvas, which
>> janks apps which have to process lots of large images. Running a canvas in
>> a worker could possibly help, but that would require both Blob and Image to
>> be transferable to be useful.
>>
>>
>>
>> Ashley Gullen
>>
>> Scirra.com
>>
>>
>>
>>
>>
>
>