Re: [whatwg] Adding features needed for WebGL to ImageBitmap

To respond on the topic of WebGL/ImageBitmap integration - and in
particular some of the features requested earlier in the thread. Apologies
if I missed a post where this stuff was already addressed directly; I
couldn't follow this thread easily because of how much context was stripped
out of replies:

Having control over when or where colorspace conversion occurs would be
tremendously valuable. Right now the only place where you have control over
this is in WebGL, and when it comes to canvas each browser seems to
implement it differently. This is already a problem for people trying to do
image processing in JavaScript; an end-user of my compiler ran into this by
writing a simple app that read pixel data out of PNGs and then discovered
that every browser had its own unique interpretation of what a simple
image's data should look like when using getImageData:

https://bugzilla.mozilla.org/show_bug.cgi?id=867594

Ultimately the core here is that without control over colorspace
conversion, any sort of deterministic image processing in HTML5 is off the
table, and you have to write your own image decoders, encoders, and
manipulation routines in JavaScript using raw typed arrays. Maybe that's
how it has to be, but it would be cool to at least support basic variations
of these use cases in Canvas since getImageData/putImageData already exist
and are fairly well-specified (other than this problem, and some nits
around source rectangles and alpha transparency).

Out of the features suggested previously in the thread, I would immediately
be able to make use of control over colorspace conversion and an ability to
opt into premultiplied alpha. Not getting premultiplied alpha, as is the
case in virtually every canvas implementation I've tried, has visible
negative consequences for image quality and also reduces the performance of
some use cases where bitmap manipulation needs to happen, due to the fact
that premultiplied alpha is the 'preferred' form for certain types of
rendering and the math works out better. I think the upsides to getting
premultiplication are the same here as they are in WebGL: faster
uploads/downloads, better results, etc.

I understand the rationale behind gregg's suggestion for flipY, but
ultimately don't know if that one makes any sense in a HTML5 context. It
basically only exists because of the annoying disagreement between APIs
like OpenGL and other APIs like HTML5 Canvas or Direct3D, specifically
about which direction the Y axis goes. Normally one would assume that you
can correct this by simply inverting heights/y coordinates in the correct
places, but when you're rendering to offscreen surfaces, the confusion over
the Y axis ends up causing you to have to do a bunch of weird things to
coordinates and sampling in order to get correct results, because your
offscreen surfaces are *actually* upside down. It's gross.

To clearly state what would make ImageBitmap useful for the use cases I
encounter and my end-users encounter:
ImageBitmap should be a canonical representation of a 2D bitmap, with a
known color space, known pixel format, known alpha representation
(premultiplied/not premultiplied), and ready for immediate rendering or
pixel data access. It's okay if it's immutable, and it's okay if
constructing one from an <img> or a Blob takes time, as long as once I have
an ImageBitmap I can use it to render and use it to extract pixel data
without user configuration/hardware producing unpredictable results.

Colorspace conversion would allow me to address outstanding bugs that
currently require my end users to manually strip color profiles and gamma
from their image files, and premultiplied alpha would dramatically improve
the performance of some test cases and shipped games out there based on my
compiler. (Naturally, this all requires browser vendors to implement this
stuff, so I understand that these gains would probably not manifest for
years.)

-kg

Received on Thursday, 18 July 2013 07:35:58 UTC