Re: [w3ctag/design-reviews] Canvas 2D color management (#646)

Thank you so much for the quick look!

Something I should have emphasized is that CanvasColorSpaceProposal.md is what was brought to WhatWG, and then the WhatWG PR is what came out of that review. It may be that I should update CanvasColorSpaceProposal.md to reflect those changes.

> Hi @ccameron-chromium!
> 
> Fantastic to see a proposal to color manage Canvas, and extend it beyond sRGB. 👍🏼
> It's unfortunate that sRGB has to be the default, but completely understandable for web compat.
> 
> Here are some questions that occurred to me on initial reading.
> 
> * You say the target is Chrome 92. However to my knowledge, there are no plans to implement `color(display-p3)` or `lab()` or `lch()` in Chrome 92. Without those, it would be impossible to _draw_ graphics that utilize P3 colors, and thus the only value-add of this implementation would be the ability to paint P3 images on canvas. Is that the case? Does Chromium plan to implement `color(display-p3)` but only for Canvas? Something else?

Indeed Chrome 92 will not have color(display-p3) et al. WCG content can be drawn to a 2D canvas via Images and via ImageData.

When we were trying to decide which pieces to pick off first (CSS color vs 2D canvas), the balance came out in favor of canvas, for applications that wanted to ensure that their images weren't crushed to sRGB (even if all CSS colors were still limited to sRGB). Ultimately both are much more useful with each other.

> * I imagine eventually we'd want to extend this to the Paint API’s [`PaintRenderingContext2D`](https://drafts.css-houdini.org/css-paint-api-1/#2d-rendering-context). Given that the context for that is pre-generated, how would that look like?

For PaintRenderingContext2D, the actual output color space is not observable by Javascript (getImageData isn't exported). This is unlike CanvasRenderingContext2D, where the color space is observable (and has historically been a fingerprinting vector). Because of that, my sense is that the user agent should be able to select the best color space for the display device (just as it does for deciding the color space in which <img> elements are drawn and composited), and potentially change that space behind the scenes. Having the application specifying a color space for PaintRenderingContext2D feels like an unnatural constraint.

Similarly, ImageBitmap and ImageBitmapRenderingContext don't want color spaces -- one should just be able to create an ImageBitmap from a source and send it to ImageBitmapRenderingContext and, by default, have it appear the same as the source would have if drawn directly as an element. (Of note is that we will likely add a color space to ImageBitmapOptions to allow asynchronous-ahead-of-time conversion for when uploading into a WebGL/GPU texture, but that is outside of the 2D context).

> * Is this intended to become 10 bit by default [once 10 bits per component](https://github.com/w3c/ColorWeb-CG/blob/master/hdr_html_canvas_element.md#higher-bit-storage-formats-for-2d-contexts) are supported? Could this introduce web compat problems?

Indeed for non-srgb-or-display-p3 spaces, we may want to default to something more than 8 bits per pixel. That's part of why we decided not to include rec2020 in the spec (the other part being disputes about its proper definition!!).

For srgb and display-p3, the overwhelming preference is for 8 bits per pixel, and so the default of 8 bits per pixel will be what we will want to stay with (using more than 8 bits per pixel comes with substantial power and memory penalties, for almost no perceptual gain). As you noted, in the HDR spec, we may want to make a selection of color space imply a particular pixel format (I'm still on the fence about that -- fortunately we're avoiding being affected by how that decision lands -- display-p3 is the most requested space).

> * > Input colors (e.g, fillStyle and strokeStyle) follow the same interpretation as CSS color literals, regardless of the canvas color space.
>   
>    What happens when someone paints an e.g. `rec2020` color on an sRGB or Display P3 canvas? Is the result gamut mapped? If so, how?

The input colors (like other inputs) [are converted](https://html.spec.whatwg.org/multipage/canvas.html#colour-spaces-and-colour-correction) from the input's color space to the canvas's color space using [relative colorimetric mapping](https://drafts.csswg.org/css-color/#valdef-color-profile-rendering-intent-relative-colorimetric), which is the "don't do anything fancy" mapping. In your example, the rec2020 color can always be transformed to some pixel in sRGB, but that pixel may have RGB values outside of the 0-to-1 interval. Relative colorimetric intent just clamps the individual color values to 0-to-1.

This is what happens today in all browsers if the browser, e.g, loads a rec2020 image that uses the full gamut and attempts to display it on a less capable monitor.

(Somewhat relatedly, one thing that came up in a separate review is that it might be useful for developer tools to have a "please pretend I have a less capable monitor than I do" mode).

> * If I understand the explainer correctly, this means that the first script that calls `getContext()` gets to define the color space the canvas is in. What happens on any subsequent `getContext()` calls, either without a colorSpace argument, or with a different one? Do they produce an error or silently return the existing context, color managed with a different color space than the one the author specified? Do they clear the contents?

The current behavior is that the subsequent call to getContext('2d') will [return the previously created context](https://html.spec.whatwg.org/multipage/canvas.html#dom-canvas-getcontext), even if it has different properties than what was requested the second time around. This applies to all of the settings (alpha, etc).

> Not sure any of these options is better than making `colorSpace` be mutable (which would also address `PaintRenderingContext2D`). It is not that unheard of to change the color space of color managed graphics contexts, e.g. it's possible in every color managed graphics application I know of, and there are several reasonable ways to do it.

Yes, this was another tricky area. There was some discussion around making the colorSpace be a mutable attribute, but there were a few things pushing against it. One was that there were indeed many reasonable things to do (clear the canvas, reinterpret_cast the pixels, convert the pixels?), and no single option was a clear winner. Another was that this matched the behavior for alpha (which will likely match the future canvas bit depth). Another was that it felt conceptually like a bad fit (especially in comparison with, e.g, WebGPU, where the GPUSwapChainDescriptor is the natural spot, and can be changed on frame boundaries).

So that's how we ended up landing where we did. Does that feel reasonable to you too?

In practice, if one wants to swap out a canvas for a differently-configured canvas, one can create the new element (or offscreen canvas) and drawImage the previous canvas into it (which will achieve the "convert" behavior).

We also briefly discussed if it was possible for the canvas to automatically update its color space to support whatever is drawn into it (turns out it's not, at least not without scrutinizing every pixel of every texture that gets sent at it, and even then that may not be desirable).

> * Am I reading it correctly that `getImageData()` will return sRGB data even in a P3 canvas, unless P3 data is explicitly requested? What's the rationale for not defaulting to the current color space?

Yes, this is a good point -- the WhatWG review changed this behavior (again, sorry I wasn't more clear about that earlier). 

The text that landed is what you suggest (getImageData returns the canvas's color space). Critically, getImageData, toDataURL, and toBlob have the property if that one exports a canvas into a (imagedata, blob, url), and then draws the result on the same canvas, the operation is a no-op (no data is lost ... unless you choose lossy compression).

> * > The color space is then an immutable property of the CanvasRenderingContext2D.
>   
>    Unless I missed it, none of your Web IDL snippets include this readonly attribute. I assume in unsupported color spaces this attribute will be `"srgb"`?

Following alpha's pattern, it's query-able using getContextAttributes (it will be in the returned [CanvasRenderingContext2DSettings](file:///Users/ccameron/chrome-dev/github/html-build/output/multipage/canvas.html#dom-context-2d-canvas-getcontextattributes)).

When [creating a context](file:///Users/ccameron/chrome-dev/github/html-build/output/multipage/canvas.html#2d-context-creation-algorithm), the color space for the context is set to the color space in the attributes, so all enum values that get past the IDL must be supported for 2D canvas and for ImageData. (Also, the proposal document advertised a feature detection interface, which was nixed in WhatWG review).

If the browser doesn't support this feature at all, then there will be no colorSpace entry in CanvasRenderingContext2DSettings, so the feature may be detected through that mechanism.

Thank you again for the quick feedback!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/w3ctag/design-reviews/issues/646#issuecomment-859135275