Re: Async Image -> ImageData conversion

I'm not sure having a hub is worth the implementation pain.

It looks like there are three pretty different things here:

ImageBitmap - Immutable images for drawing on canvas. Typically
premultiplied data located on the GPU
ImageData - Mutable script accessible array of data. Unpremultiplied
data located in main memory
ImageElement/Blob - A pointer to some compressed image data.

Unpremultiplied -> premultiplied is a lossy conversion step which can
cause additional implementor/user pain.
i.e. if you have an ImageData and you want to make it into a blob,
it's not clear if the browser will
do unpremul -> premul -> unpremul encoded images or unpremul ->
unpremul encoded.

It's seems like the easiest thing for users and implementors is just
to add the necessary functions in all
of the important places:

The important ones of these are:
Blob -> ImageData
HTMLImageElement -> ImageData
ImageData -> Blob

There are a bunch of conversions that aren't covered directly by these but I'm
not sure it makes us much sense to short circut them beyond what functionality
is currently exposed. e.g.
Going HTMLVideoElement -> ImageBitmap -> Blob
isn't really much worse than doing:
HTMLVideoElement -> CanvasRenderContext2D -> Blob

-Jeff

On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad <junov@google.com> wrote:
> Making ImageData the "hub" would imply more copies in memory in many cases.
> Because ImageData is mutable (not to mention the alpha multiplication issues
> which are also a factor), it cannot share its image buffer with the source
> it was created from, unlike ImageBitmap.  Immutability is a significant
> advantage of ImageBitmap, which allows for zero-copy code paths in many
> cases, which helps with both performance and memory consumption.
>
> On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen <ashley@scirra.com> wrote:
>>
>> If it's difficult to make ImageBitmap both an efficient drawing source and
>> a conversion intermediary, then we could add conversion functions to each
>> object that can be converted, e.g.:
>>
>> HTMLImageElement.toBlob()
>> HTMLImageElement.toImageData()
>> Blob.toImageData()
>> ImageData.toBlob()
>>
>> This seems inconsistent. For example to convert a Blob to an Image, you
>> should create a URL and set an Image's src to it; to convert a Blob to an
>> ImageBitmap, you should use createImageBitmap(blob); and then under this
>> proposal there's a third approach to convert a Blob to an ImageData, using
>> Blob.toImageData(). It also has a larger surface area, requiring changes to
>> several interfaces.
>>
>> So perhaps it would be better to make ImageData the "hub" of conversion,
>> more like the first proposal I made. If ImageBitmap is a GPU-hosted
>> premultiplied resource, then ImageData is an expressly not-on-GPU
>> not-premultiplied alternative. That would mean adding something like this to
>> ImageData:
>>
>> typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
>>
>> partial interface ImageData {
>>     static Promise<ImageData> create(ImageDataSource source)
>>     Promise<Blob> toBlob()
>> }
>>
>> ImageData can also be converted to HTMLImageElement via toBlob,
>> ImageBitmap via createImageBitmap, or a canvas via putImageData (which is
>> still synchronous, but no longer needs to be done purely for conversion
>> reasons, so probably means you really want it to appear in the canvas and
>> therefore should remain synchronous).
>>
>> This covers all the conversion use cases I can think of without
>> complicating ImageBitmap's "without undue latency" requirement. There's no
>> more transferring going on either, which I think is now unnecessary since
>> you can get from HTMLImageElement to ImageData with one call. I think it's
>> more future-proof too since future types can be added to ImageDataSource
>> allowing new types to be converted without a new method being added.
>>
>> Does this approach sound better?
>>
>> Ashley
>>
>>
>> On 26 June 2015 at 16:37, Boris Zbarsky <bzbarsky@mit.edu> wrote:
>>>
>>> On 6/26/15 4:07 AM, Ashley Gullen wrote:
>>>>
>>>> I don't think we should extend HTMLImageElement because it is not
>>>> available in workers. Adding the conversion methods to ImageBitmap
>>>> allows workers to perform conversions using Blob (compressed image data)
>>>> in the place of HTMLImageElement.
>>>
>>>
>>> Maybe I wasn't clear.  I was suggesting that we have the methods on both
>>> HTMLImageElement and ImageBitmap (and possibly on any other things we feel
>>> should have the methods directly).
>>>
>>>> I like the suggestion that "ImageBitmap be the hub of image conversion",
>>>
>>>
>>> I agree that it sounds appealing, but it means ImageBitmap now has to
>>> serve two masters: it has to be something that you can paint from quickly
>>> (premultiplied, probably lives on the GPU) _and_ it needs to be something
>>> you can transferToImageData efficiently (better to not live on the GPU for
>>> this).
>>>
>>> Maybe that's OK; it's just a bit of a warning flag from my point of view
>>> when a single object is meant to do multiple quite different things; it
>>> makes it harder to have it be good at all of them...
>>>
>>> -Boris
>>
>>
>

Received on Friday, 3 July 2015 00:17:15 UTC