Re: Adding support High Dynamic Range (HDR) imagery to HTML Canvas: a baseline proposal

On Wed, Jun 28, 2023 at 6:34 PM Dale Curtis <dalecurtis@google.com> wrote:

> On the Chrome side we've been in contact with +Christopher Cameron
> <ccameron@google.com> on this. The proposal looks reasonable so far. In
> terms of things salient to the Media WG, we should:
> - Ensure VideoColorSpace properly converts when drawn into a HDR canvas.
> Specifically that the outputs of video, VideoDecoder, and ImageDecoder map
> correctly into destination spaces.
>

[See below, there is no standard behavior when drawing *existing* HDR
content to *existing* SDR canvases].


> - Ensure high bit depth pixel formats work properly:
> https://github.com/w3c/webcodecs/issues/384#issuecomment-1169230721
>

My main feedback is to try to slice this into smaller pieces where
possible. In particular:

*Slice 1: Separate out 2D canvas >8 bit support*

There is already a separate proposal
<https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md> and github
issue <https://github.com/whatwg/html/issues/8708>.


It's very specific to 2D canvas. WebGL and WebGPU already support enough
texture, renderbuffer, and framebuffer formats to make this proposal
valuable without this part.

It's not clear that float16 is the best fit for the proposal here (unorm16
or rgb10a2unorm may be better if we're targeting HLG and PQ).


*Slice 2: Solve the problem of rendering HLG and PQ content to SDR as a
prerequisite*

See this issue <https://github.com/whatwg/html/issues/9112> and this canvas
test page
<https://ccameron-chromium.github.io/webgl-examples/hdr-on-canvas.html>
(Mozilla,
Safari, and Chrome all do different things).

This means defining how to do HLG->SDR and PQ->SDR conversion. We should
enumerate what metadata needs to be supported, and how it feeds into this
conversion (ISO 22028-5 lists MDCV, CLLI, CCV, NDLW, and REVE as options).


We should add WPT tests to enforce uniform behavior across the various
browsers.

This is the "From HDR to SDR conversion
<https://github.com/w3c/ColorWeb-CG/blob/8eac62f15e5b41940636beb031f6214b61e933af/hdr_html_canvas_element.md#from-hdr-to-sdr-color-spaces>"
part of the proposal. It's a delicate enough topic that I think it merits
its own separate discussion.


From there, I think that we can separate this feature further into two
parts:

*Slice 3: Add rec2100-hlg and rec2100-pq to PredefinedColorSpace
<https://html.spec.whatwg.org/multipage/canvas.html#predefinedcolorspace>*

This automatically adds support to 2D canvas (via
CanvasRenderingContext2DSettings
<https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/getContextAttributes>),
ImageData (via colorSpace
<https://developer.mozilla.org/en-US/docs/Web/API/ImageData/colorSpace>),
WebGL texture import (via unpackColorSpace
<https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/unpackColorSpace>),
WebGL rendering (via drawingBufferColorSpace
<https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/drawingBufferColorSpace>),
WebGPU texture import (via GPUImageCopyTextureTagged
<https://webgpu.rocks/reference/dictionary/gpuimagecopytexturetagged/>),
and WebGPU rendering (via GPUCanvasConfiguration
<https://webgpu.rocks/reference/dictionary/gpucanvasconfiguration/#idl-gpucanvasconfiguration>)
in one API change. So it's A Big Deal.

This requires defining PQ->HLG and HLG->PQ conversions. This is covered in
the "Between HDR color spaces
<https://github.com/w3c/ColorWeb-CG/blob/8eac62f15e5b41940636beb031f6214b61e933af/hdr_html_canvas_element.md#between-hdr-color-spaces>"
section, and I think is in good shape. It does need careful scrutiny,
though. E.g, what do we do with the various HDR metadatas on both the
source and the destination there?

It also requires defining SDR->PQ and SDR->HLG conversions. This is covered
in the "From SDR to HDR color spaces
<https://github.com/w3c/ColorWeb-CG/blob/8eac62f15e5b41940636beb031f6214b61e933af/hdr_html_canvas_element.md#from-sdr-to-hdr-color-spaces>"
section (I'm not completely sold on the proposed math -- more research is
needed).


*Slice 4: Add an API for specifying HDR metadata to the various canvas
rendering context APIs*

This is in the "Add HDR rendering behavior and HDR metadata to
CanvasRenderingContext2DSettings
<https://github.com/w3c/ColorWeb-CG/blob/8eac62f15e5b41940636beb031f6214b61e933af/hdr_html_canvas_element.md#add-hdr-rendering-behavior-and-hdr-metadata-to-canvasrenderingcontext2dsettings>"
section.

I would suggest changing the API to have an "HDRMetadata" structure, which
has optional entries for each type of metadata. We should also indicate how
it would be specified in WebGL and WebGPU.

We want to ensure that this aligns with WebCodecs (both for images and
video). This should probably include the various ISO 22028-5 metadatas
(MDCV, CLLI, CCV, NDLW, and REVE).


The "HDRMetadata" dictionary should also have an "extended range" entry
that says "do not perform any OS-level tonemapping to me -- clamp any
out-of-range values" (this is a common request).


I do feel quite strongly that (1) and (2,3,4) really need to be completely
separated out.

Given my druthers, I would swarm this problem by independently attacking
(1), (2-then-3), and (4-with-just-extended-range-and-then-the-rest-after-3)..

On Tue, Jun 27, 2023 at 9:33 PM Pierre-Anthony Lemieux <pal@sandflow.com>
> wrote:
>
>> (The message was intended for the MEIG... but we also wanted to get
>> the Media WG's feedback, so below is the equivalent message for this
>> group. Apologies for the confusion, I blame autocorrect.)
>>
>> Dear Media WG:
>>
>> The Color on the Web CG [1] would very much appreciate your feedback
>> on the following strawman, which motivates and proposes baseline
>> modifications to the HTML Canvas API to enable support for
>> High-Dynamic Range (HDR) imagery:
>>
>>
>> https://github.com/w3c/ColorWeb-CG/blob/8eac62f15e5b41940636beb031f6214b61e933af/hdr_html_canvas_element.md
>>
>> The proposed modifications allow the HTML Canvas API to manipulate
>> High Dynamic Range (HDR) images expressed using the widespread BT.2100
>> PQ and BT.2100 HLG color spaces, without precluding adding future HDR
>> capabilities, such as support for additional color spaces like a
>> linear high-dynamic range color space.
>>
>> Feedback can be provided at [2].
>>
>> Best,
>>
>> -- Color on the Web CG chairs
>>
>> [1] https://www.w3.org/community/colorweb/
>> [2] https://github.com/w3c/ColorWeb-CG/issues
>>
>>
>>
>> On Tue, Jun 27, 2023 at 9:17 PM Pierre-Anthony Lemieux <pal@sandflow.com>
>> wrote:
>> >
>> > Good morning/evening,
>> >
>> > Following-up on my presentation on April 25,  the Color on the Web CG
>> > [1] would very much appreciate any feedback the MEIG might have on the
>> > following strawman, which motivates and proposes baseline
>> > modifications to the HTML Canvas API to enable support for
>> > High-Dynamic Range (HDR) imagery:
>> >
>> >
>> https://github.com/w3c/ColorWeb-CG/blob/8eac62f15e5b41940636beb031f6214b61e933af/hdr_html_canvas_element.md
>> >
>> > The proposed modifications allow the HTML Canvas API to manipulate
>> > High Dynamic Range (HDR) images expressed using the widespread BT.2100
>> > PQ and BT.2100 HLG color spaces, without precluding adding future HDR
>> > capabilities, such as support for additional color spaces like a
>> > linear high-dynamic range RGB color space.
>> >
>> > Best,
>> >
>> > -- Pierre [CG co-chair]
>> >
>> > [1] https://www.w3.org/community/colorweb/
>>
>>