Re: Async Image -> ImageData conversion

On Thu, Jun 25, 2015 at 9:40 AM, Justin Novosad <junov@google.com> wrote:

>
>
> On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen <ashley@scirra.com> wrote:
>
>> I see that OffscreenCanvas also specifies a close() method for
>> ImageBitmap. Perhaps browsers could use copy-on-write with the
>> toImageData() method? Then you can efficiently either transfer or copy
>> depending on if you close the ImageBitmap you transferred to an ImageData.
>>
>
>> Transfer:
>> createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close()
>> No copy made, contents transferred to the imageData
>>
>> createImageBitmap() -> imageBitmap.toImageData() -> access resulting
>> imageData.data
>> Copy made on first access
>>
>
> Copy on write is probably not a good idea because it would require
> injecting an observer pattern (or some kind of notification mechanism) into
> Uint8ClampedArray data assignment, which would realistically degrade
> performance of Image processing pixel loops.  Having a one step transfer
> API (transferToImageData) avoids adding overhead to what is otherwise just
> a simple memory access.
>

Something more: Blink, and probably other implementations as well, stores
decoded images in alpha pre-multiplied form.  ImageData is not
pre-multiplied.  For this reason sharing a buffer between ImageBitmap and
ImageData would be somewhat inconvenient.


>>
>> On 24 June 2015 at 21:54, Kenneth Russell <kbr@google.com> wrote:
>>
>>> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen <ashley@scirra.com>
>>> wrote:
>>> > Sorry for the confusion. Yes, the latest URL is:
>>> >
>>> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
>>> > I'm new to specs and WebIDL, my intent was to say those are new
>>> methods on
>>> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say
>>> that?
>>> > (I updated the URL to say that instead)
>>> >
>>> > The wording of "undue latency" in the ImageBitmap spec does not appear
>>> to
>>> > rule out ImageBitmap storing its encoded image data, and lazily
>>> decoding it.
>>> > However I'm not sure what the point of that would be, since it seems
>>> to be
>>> > how drawing HTMLImageElements to a canvas already works. I see a
>>> couple of
>>> > options:
>>> > - allow lazy decoding in ImageBitmap, and therefore converting to
>>> ImageData
>>> > is an explicit request to decompress
>>> > - require ImageBitmap to decompress its contents, and make it available
>>> > through a read-only Uint8ClampedArray like ImageData has. Therefore,
>>> > converting to ImageData indicates the intent to modify this content,
>>> > therefore a copy is warranted.
>>> > - provide a way to "neuter" the ImageBitmap when converting it to
>>> ImageData,
>>> > transferring its contents and avoiding any copy, which is useful if the
>>> > ImageBitmap is a temporary object only being created for the purposes
>>> of
>>> > getting the ImageData. I guess this would be similar to transferrable
>>> > objects with workers.
>>>
>>> Perhaps this could be done by specifying a "transferToImageData" method.
>>>
>>> This proposal makes ImageBitmap neuterable, and there's intent to
>>> implement it in order to allow rendering to canvas contexts from
>>> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas .
>>>
>>> -Ken
>>>
>>>
>>> > - add some kind of "load" method on ImageBitmap. It may store its
>>> contents
>>> > in compressed form, but calling "load" causes it to be decompressed (or
>>> > loaded from somewhere else). The ImageBitmap is only guaranteed to be
>>> drawn
>>> > with "undue latency" after the "load" call. Therefore for the use case
>>> of
>>> > low-latency rendering "load" can always be called immediately after
>>> > creation, but for conversion purposes not calling "load" avoids
>>> duplicating
>>> > memory.
>>> >
>>> > Note for devices with discrete GPUs, it's possible that ImageBitmap
>>> stores
>>> > the decompressed image data in a GPU texture but does not have it in
>>> system
>>> > RAM. In this case making a copy for the ImageData is also warranted.
>>> >
>>> > I'm a web developer proposing this because it would be useful for my
>>> > purposes, I've no idea which of these would be favoured by
>>> implementors or
>>> > the spec process. I'm happy to get involved in spec work though so
>>> please
>>> > let me know if you have any feedback/suggestions on anything I'm doing
>>> here.
>>> >
>>> > Ashley
>>> >
>>> >
>>> >
>>> > On 24 June 2015 at 19:12, Boris Zbarsky <bzbarsky@mit.edu> wrote:
>>> >>
>>> >> On 6/19/15 5:43 AM, Ashley Gullen wrote:
>>> >>>
>>> >>> I've not done this before, so I've no idea if this is the
>>> right/useful
>>> >>> approach, but I drafted a spec for it here:
>>> >>>
>>> >>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>>> >>
>>> >>
>>> >> Ashley,
>>> >>
>>> >> We at Mozilla were just discussing this proposal, and we have a
>>> concern.
>>> >>
>>> >> With this proposal, going from an <img> to an ImageData requires
>>> >> conversion of an intermediate ImageBitmap.  Unfortunately, an
>>> ImageBitmap is
>>> >> specified to be paintable "without undue latency", which we interpret
>>> to
>>> >> mean it needs to have decoded image data, and the ImageData will end
>>> up
>>> >> needing to copy said data in practice (because it needs an editable
>>> copy).
>>> >>
>>> >> That creates two copies of the decoded image data in memory, which
>>> seems
>>> >> fairly undesirable on memory-constrained devices.
>>> >>
>>> >> -Boris
>>> >>
>>> >>
>>> >
>>>
>>
>>
>