Re: Async Image -> ImageData conversion

On 2 July 2015 at 21:36, Jeff Muizelaar <jmuizelaar@mozilla.com> wrote:
>
> The important ones of these are:
> Blob -> ImageData
> HTMLImageElement -> ImageData
> ImageData -> Blob
>

These conversions are covered by what I proposed in my previous email, so I
drafted another spec with them:
https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html

I also mentioned canEncodeType and canDecodeType in another email thread,
but I think they are important to the same use cases, so I've put them in
the draft as well to provoke discussion.

Ashley








>
> There are a bunch of conversions that aren't covered directly by these but
> I'm
> not sure it makes us much sense to short circut them beyond what
> functionality
> is currently exposed. e.g.
> Going HTMLVideoElement -> ImageBitmap -> Blob
> isn't really much worse than doing:
> HTMLVideoElement -> CanvasRenderContext2D -> Blob
>
> -Jeff
>
> On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad <junov@google.com> wrote:
> > Making ImageData the "hub" would imply more copies in memory in many
> cases.
> > Because ImageData is mutable (not to mention the alpha multiplication
> issues
> > which are also a factor), it cannot share its image buffer with the
> source
> > it was created from, unlike ImageBitmap.  Immutability is a significant
> > advantage of ImageBitmap, which allows for zero-copy code paths in many
> > cases, which helps with both performance and memory consumption.
> >
> > On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen <ashley@scirra.com>
> wrote:
> >>
> >> If it's difficult to make ImageBitmap both an efficient drawing source
> and
> >> a conversion intermediary, then we could add conversion functions to
> each
> >> object that can be converted, e.g.:
> >>
> >> HTMLImageElement.toBlob()
> >> HTMLImageElement.toImageData()
> >> Blob.toImageData()
> >> ImageData.toBlob()
> >>
> >> This seems inconsistent. For example to convert a Blob to an Image, you
> >> should create a URL and set an Image's src to it; to convert a Blob to
> an
> >> ImageBitmap, you should use createImageBitmap(blob); and then under this
> >> proposal there's a third approach to convert a Blob to an ImageData,
> using
> >> Blob.toImageData(). It also has a larger surface area, requiring
> changes to
> >> several interfaces.
> >>
> >> So perhaps it would be better to make ImageData the "hub" of conversion,
> >> more like the first proposal I made. If ImageBitmap is a GPU-hosted
> >> premultiplied resource, then ImageData is an expressly not-on-GPU
> >> not-premultiplied alternative. That would mean adding something like
> this to
> >> ImageData:
> >>
> >> typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
> >>
> >> partial interface ImageData {
> >>     static Promise<ImageData> create(ImageDataSource source)
> >>     Promise<Blob> toBlob()
> >> }
> >>
> >> ImageData can also be converted to HTMLImageElement via toBlob,
> >> ImageBitmap via createImageBitmap, or a canvas via putImageData (which
> is
> >> still synchronous, but no longer needs to be done purely for conversion
> >> reasons, so probably means you really want it to appear in the canvas
> and
> >> therefore should remain synchronous).
> >>
> >> This covers all the conversion use cases I can think of without
> >> complicating ImageBitmap's "without undue latency" requirement. There's
> no
> >> more transferring going on either, which I think is now unnecessary
> since
> >> you can get from HTMLImageElement to ImageData with one call. I think
> it's
> >> more future-proof too since future types can be added to ImageDataSource
> >> allowing new types to be converted without a new method being added.
> >>
> >> Does this approach sound better?
> >>
> >> Ashley
> >>
> >>
> >> On 26 June 2015 at 16:37, Boris Zbarsky <bzbarsky@mit.edu> wrote:
> >>>
> >>> On 6/26/15 4:07 AM, Ashley Gullen wrote:
> >>>>
> >>>> I don't think we should extend HTMLImageElement because it is not
> >>>> available in workers. Adding the conversion methods to ImageBitmap
> >>>> allows workers to perform conversions using Blob (compressed image
> data)
> >>>> in the place of HTMLImageElement.
> >>>
> >>>
> >>> Maybe I wasn't clear.  I was suggesting that we have the methods on
> both
> >>> HTMLImageElement and ImageBitmap (and possibly on any other things we
> feel
> >>> should have the methods directly).
> >>>
> >>>> I like the suggestion that "ImageBitmap be the hub of image
> conversion",
> >>>
> >>>
> >>> I agree that it sounds appealing, but it means ImageBitmap now has to
> >>> serve two masters: it has to be something that you can paint from
> quickly
> >>> (premultiplied, probably lives on the GPU) _and_ it needs to be
> something
> >>> you can transferToImageData efficiently (better to not live on the GPU
> for
> >>> this).
> >>>
> >>> Maybe that's OK; it's just a bit of a warning flag from my point of
> view
> >>> when a single object is meant to do multiple quite different things; it
> >>> makes it harder to have it be good at all of them...
> >>>
> >>> -Boris
> >>
> >>
> >
>