[whatwg] Canvas and color colorspaces (was: WebGL and ImageBitmaps)

tl;dr: The color space of canvas backing stores is undefined, which causes
problems for many web devs, but also has non-negligible advantages. So be
careful what you wish for.

I saw some confusion and questions needing answers in the "WebGL and
ImageBitmaps" thread regarding color management. I will attempt to clarify
to the best of my abilities. Though I am knowledgeable on the subject, I am
not an absolute authority, so others are welcome to correct me if I am
wrong about anything.

Color management... To make a long story short, there are two types of
color profiles : input profiles and output profiles for characterizing
input devices (cameras, scanners) and output devices (displays, printers)
respectively.
Image files will usually encode their color information in a standard color
space or in a an input device dependent space. If colors are encoded in a
color space that is different from the format's default, then a color
profile or a color space identifier must be encoded into the image
resource's metadata.

To present color-managed image content on screen, the image needs to be
converted from whatever color space the image was encoded into into a
standard "connection space" using the color profile or color space metadata
from the image resource. Then the colors need to be converted from the
profile connection space to the output space, which is provided by the
OS/display driver. Depending on the OS and hardware configuration, the
output space may be a standard color space (like sRGB), or a
device-specific color profile.

Currently, many color-managed software applications rely on the codec to
take care of the entire color-management process for image and video
content, meaning that the decoded image data is in output-referred color
space (i.e. the display's profile was applied).  There are practical
reasons for this, the most important ones being color fidelity and memory
consumption.  Let me explain. The profile connection space is typically CIE
XYZ or CIE L*a*b. I wont get into the technical details of how these work
except to say that they are device independent and allow for an accurate
representation of the whole spectrum of human-visible colors. This makes it
possible to map colors from a wide gamut camera to a wide gamut display
with high color fidelity for all the colors that are located in the
intersection of the color gamuts of both the input and output devices. If
we were forced to convert the image to an intermediate sRGB representation,
the colors in the image would be clamped to the sRGB gamut (which is
narrower than the gamuts of many devices). Currently, most browsers avoid
doing that for <img>, and therefore provide (more or less) optimal image
and video color fidelity for users of wide gamut devices. Also, an
intermediate representation in 8-bit sRGB means loss of precision due to
rounding errors, as opposed to the profile connection space which uses
higher precision registers for intermediate color values to avoid precision
issues caused by rounding.  To avoid perceptible precision issues in an
intermediate sRGB representation, we'd have to increase the bit depth and
therefore use more RAM for storing decoded image data.

All of this is to say that there are good reasons for the current situation
where we deal with decoded images that have the output device's color
profile pre-applied: color fidelity and memory consumption.

In the case of 2D canvas, the color space for the backing store is
unspecified, and many implementations have chosen to use the output
device's color space, which has many advantages:
* images and videos are already decoded directly into that space
* no color conversion is necessary when presenting the canvas on screen
(good for performance)
* there is no loss of precision due the use of a limited-precision
intermediate color space.
* the color gamut is not constrained by an intermediate color space (like
sRGB).
And disadvantages:
* Compositing operations produce incorrect results because most of them
(including source-over) are affected by the color space.
* direct pixel manipulation using put/getImageData exposes data in a color
space that is undefined, making it extremely challenging to perform many
types of image processing and image generation tasks in a
device-independent way.
* The device-dependent behavior of a drawImage/getImageData round trip is a
known fingerprinting vector.

Right now, I am hearing a lot of complaints regarding the lack of a
standardized color space for canvases, and in particular the impact this
has on applications that try to do cool things with put/getImageData, or
generate images procedurally.  I want to make sure everyone understands
there is a trade-off to fixing this, so be careful what you wish for.

I am especially concerned about the issue of color gamut clamping, which
will increase in relevance as wide gamut devices become more widespread.
 If we were to decide that canvas backing stores had to be in sRGB, that
would mean that a wide gamut image viewed on a wide gamut display would
look best when displayed in an <img> and would be duller when drawn through
a <canvas>, whether it be 2D or WebGL.  Is that something we are willing to
live with in the name of standardizing the color space of ImageData?  I
must admit I was in favor of moving canvases to sRGB until I reviewed some
of Noel Gordon's recent work which brought the gamut and precision issues
to my attention.

Rik: to answer your question about your experiment: there is no issue with
a put/getImageData round trip. You will get back the same color you put in
(at least for opaque colors, but that is another story).  The issue is with
a drawImage/getImageData round trip.  For the same source image,
getImageData will return different values depending on the display profile
of the system you are running on.

Ken: You mentioned to me off thread that built-in support for sRGB render
buffers in OpenGL ES 3 may make it easier to move rendering to sRGB on next
gen devices. I tend to agree, and I think it will also mitigate the loss of
precision issue, but it still implies clamping wide gammut media to the
sRGB range.

    -Justin

Received on Thursday, 22 May 2014 19:22:07 UTC