Re: Async Image -> ImageData conversion

On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen <ashley@scirra.com> wrote:

> I see that OffscreenCanvas also specifies a close() method for
> ImageBitmap. Perhaps browsers could use copy-on-write with the
> toImageData() method? Then you can efficiently either transfer or copy
> depending on if you close the ImageBitmap you transferred to an ImageData.
>

> Transfer:
> createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close()
> No copy made, contents transferred to the imageData
>
> createImageBitmap() -> imageBitmap.toImageData() -> access resulting
> imageData.data
> Copy made on first access
>

Copy on write is probably not a good idea because it would require
injecting an observer pattern (or some kind of notification mechanism) into
Uint8ClampedArray data assignment, which would realistically degrade
performance of Image processing pixel loops.  Having a one step transfer
API (transferToImageData) avoids adding overhead to what is otherwise just
a simple memory access.


>
> On 24 June 2015 at 21:54, Kenneth Russell <kbr@google.com> wrote:
>
>> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen <ashley@scirra.com> wrote:
>> > Sorry for the confusion. Yes, the latest URL is:
>> >
>> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
>> > I'm new to specs and WebIDL, my intent was to say those are new methods
>> on
>> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say
>> that?
>> > (I updated the URL to say that instead)
>> >
>> > The wording of "undue latency" in the ImageBitmap spec does not appear
>> to
>> > rule out ImageBitmap storing its encoded image data, and lazily
>> decoding it.
>> > However I'm not sure what the point of that would be, since it seems to
>> be
>> > how drawing HTMLImageElements to a canvas already works. I see a couple
>> of
>> > options:
>> > - allow lazy decoding in ImageBitmap, and therefore converting to
>> ImageData
>> > is an explicit request to decompress
>> > - require ImageBitmap to decompress its contents, and make it available
>> > through a read-only Uint8ClampedArray like ImageData has. Therefore,
>> > converting to ImageData indicates the intent to modify this content,
>> > therefore a copy is warranted.
>> > - provide a way to "neuter" the ImageBitmap when converting it to
>> ImageData,
>> > transferring its contents and avoiding any copy, which is useful if the
>> > ImageBitmap is a temporary object only being created for the purposes of
>> > getting the ImageData. I guess this would be similar to transferrable
>> > objects with workers.
>>
>> Perhaps this could be done by specifying a "transferToImageData" method.
>>
>> This proposal makes ImageBitmap neuterable, and there's intent to
>> implement it in order to allow rendering to canvas contexts from
>> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas .
>>
>> -Ken
>>
>>
>> > - add some kind of "load" method on ImageBitmap. It may store its
>> contents
>> > in compressed form, but calling "load" causes it to be decompressed (or
>> > loaded from somewhere else). The ImageBitmap is only guaranteed to be
>> drawn
>> > with "undue latency" after the "load" call. Therefore for the use case
>> of
>> > low-latency rendering "load" can always be called immediately after
>> > creation, but for conversion purposes not calling "load" avoids
>> duplicating
>> > memory.
>> >
>> > Note for devices with discrete GPUs, it's possible that ImageBitmap
>> stores
>> > the decompressed image data in a GPU texture but does not have it in
>> system
>> > RAM. In this case making a copy for the ImageData is also warranted.
>> >
>> > I'm a web developer proposing this because it would be useful for my
>> > purposes, I've no idea which of these would be favoured by implementors
>> or
>> > the spec process. I'm happy to get involved in spec work though so
>> please
>> > let me know if you have any feedback/suggestions on anything I'm doing
>> here.
>> >
>> > Ashley
>> >
>> >
>> >
>> > On 24 June 2015 at 19:12, Boris Zbarsky <bzbarsky@mit.edu> wrote:
>> >>
>> >> On 6/19/15 5:43 AM, Ashley Gullen wrote:
>> >>>
>> >>> I've not done this before, so I've no idea if this is the right/useful
>> >>> approach, but I drafted a spec for it here:
>> >>>
>> >>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>> >>
>> >>
>> >> Ashley,
>> >>
>> >> We at Mozilla were just discussing this proposal, and we have a
>> concern.
>> >>
>> >> With this proposal, going from an <img> to an ImageData requires
>> >> conversion of an intermediate ImageBitmap.  Unfortunately, an
>> ImageBitmap is
>> >> specified to be paintable "without undue latency", which we interpret
>> to
>> >> mean it needs to have decoded image data, and the ImageData will end up
>> >> needing to copy said data in practice (because it needs an editable
>> copy).
>> >>
>> >> That creates two copies of the decoded image data in memory, which
>> seems
>> >> fairly undesirable on memory-constrained devices.
>> >>
>> >> -Boris
>> >>
>> >>
>> >
>>
>
>