Re: Linear canvas vs non-linear pixels.

FYI 10-bit YUV 4:2:0 is not 24bpp. V210 is a common packing for 10-bit YUV and it wastes 2 bits/32 bits.

In general, bit rate for 4:2:0 is 1/2 of 4:4:4, bit rate for 4:2:2 is 2/3 of 4:4:4, ignoring packing bits.

Lars


From: Christopher Cameron <ccameron@google.com>
Date: Wednesday, March 17, 2021 at 7:13 PM
To: Leonard Rosenthol <lrosenth@adobe.com>
Cc: Pierre Lemieux <pal@sandflow.com>, "public-colorweb@w3.org" <public-colorweb@w3.org>
Subject: Re: Linear canvas vs non-linear pixels.
Resent-From: <public-colorweb@w3.org>
Resent-Date: Wednesday, March 17, 2021 at 7:12 PM

Yes, macOS and iOS are roughly identical. Android supports float16 and unorm10 (see here<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.android.com%2Freference%2Fandroid%2Fview%2FSurfaceControl.Builder%23setFormat(int)&data=04%7C01%7Cborg%40adobe.com%7Cc04e04acfa814c64f4f608d8e9cc8453%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637516411897668143%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XURdxASSnhgZ7qYI9H57gxS0RxuU3SsfDT8FeDrQ25s%3D&reserved=0> for SurfaceControl API, which I think is the modern relevant one).

A device without a GPU won't be doing HDR canvas. I looked around for low-end phones without a GPU, and the last ones I could find were from 2009.

To the point you're making: Is having a GPU capable of float16 an onerous requirement? I don't think so. HDR cannot be done in 8 bits. GPUs that don't support float16 also don't support >8 bit colors (OpenGL ES 2.0 requires neither, OpenGL ES 3.0 requires both).

You're not wrong though -- float16 should scare you. The principal reason to avoid float16 isn't "the GPU can't handle it as a texture format" but rather "the memory bandwidth requirement is going to kill the system." There is suffering just from the fact that canvas is perforce RGB. An 8-bit YUV 4:2:0 frame is 12bits/pixel, an RGBA8 canvas is 32 bits/pixel. A 10-bit YUV 4:2:0 frame is 24bits/pixel (IIUC there's some dead space there -- haven't touched it with my hands, so I may be wrong) while an RGB10A2 canvas is is 32bits/pixel and an RGBA16F canvas is 64bits/pixel. That's where the pain will be.

I don't think we're going to be making float16 a requirement. A 10-bit-per-channel PQ or HLG format is quite reasonable, and perhaps there will be a device that will refuse to create anything but those formats (e.g: any device that has "just enough" memory bandwidth).



On Wed, Mar 17, 2021 at 1:48 AM Leonard Rosenthol <lrosenth@adobe.com<mailto:lrosenth@adobe.com>> wrote:
Chris – can you also comment (if you know) about mobile OS support?  For example, does the MacOS state also match iOS?   What about Android?   What about performance concerns of floating point operations and lack of GPU on many lower end Android devices?

Leonard

From: Christopher Cameron <ccameron@google.com<mailto:ccameron@google.com>>
Date: Wednesday, March 17, 2021 at 2:40 AM
To: Pierre-Anthony Lemieux <pal@sandflow.com<mailto:pal@sandflow.com>>
Cc: "public-colorweb@w3.org<mailto:public-colorweb@w3.org>" <public-colorweb@w3.org<mailto:public-colorweb@w3.org>>
Subject: Re: Linear canvas vs non-linear pixels.
Resent-From: <public-colorweb@w3.org<mailto:public-colorweb@w3.org>>
Resent-Date: Wednesday, March 17, 2021 at 2:39 AM



On Tue, Mar 16, 2021 at 1:50 PM Pierre-Anthony Lemieux <pal@sandflow.com<mailto:pal@sandflow.com>> wrote:
Thanks to Chris for drafting a concrete proposal [0]. Below is a
potential challenge when handling PQ HDR content.

Starting from mastering, video pixels are coded as non-linear integer
code values, e.g. HLG, PQ, etc..

These pixels ideally remain unmodified until they are rendered to the
display since each conversion carries a risk that fidelity will be
lost. This is the case even in the case where the color space and the
pixel encoded remain the same.

The strawman currently requires that PQ pixels be converted to linear
values when written to Canvas:

   PQ CV ----F----> linear value -----G-----> PQ CV

This is non-trivial and computationally expensive since F is highly
non-linear and covers many orders of magnitude. It also requires G to
be the exact inverse of F, which is not always simple if F and G are
provided by different implementations.

How do we provide a means of avoiding this complexity and potential
loss of fidelity?

Could one approach be to add optional support for non-linear pixel
encodings, e.g. add 'rec2100--pq' to [1]. In other words,
implementation would need to support 'rec2020-linear' but could also
support 'rec2100--pq'.

Adding a 'rec2100-pq' and 'rec2100-hlg' color is quite reasonable.

Support is spotty on desktop operating systems.
· Windows has a "rec2100-pq + rgb10-alpha2" mode
o DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020+DXGI_FORMAT_R10G10B10A2_UNORM
o This would be very efficient for the application you discuss
· Windows has no "rec2100-hlg" support
o That is, no support for RGB HLG buffers (there does exist YUV HLG support, of course)
· macOS had a "rec2100-pq" CGColorSpace (kCGColorSpaceITUR_2020_PQ_EOTF) but that has been deprecated
o PQ can, of course, be used with video<http://kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ>
· macOS also deprecated "rec2100-hlg" CGColorSpace (kCGColorSpaceITUR_2020_HLG) but has a "display-p3-hlg" space.
All of this is to say that the browser would, in the background, be converting into a floating point format behind the user's back most of the time. But it may be that TV-embedded browsers will have a particular signal and pixel format that is desired.

So I'm in favor of this being exposed if it's well-supported.