Fwd: half-float PNG files stored as 16-bit/component images

More "true" HDR ideas (which I would like to discuss next week).

- "True" HDR=At least 5 bit exponents (half floats). Not sure if there is a
better name.
- Generally I think there's actually a huge amount of freedom in PNG to add
true HDR images *without breaking backwards compatibility*. The format
already supports 16-bit/components and ancillary (extension) chunks, which
is crucial.
- It's possible to rotate left the bits of the half-float values (as 16-bit
uint's) more than once, and store the rotate amount in an ancillary chunk.
Legacy PNG software sees a brighter image, and new software can undo the
per-pixel bit rotation. This is entirely a lossless procedure on half-float
values.
- Any lossless transform on the half-float values that helps legacy PNG
software retrieve a better looking LDR image is usable, as long as it's
relatively simple/fast. (But which invertible function(s)?)
- Non-lossless transforms that store 16-bit values may also be useful. A
smart writer could choose the best transform that results in a decent
looking LDR image when the high byte is fetched, but a true HDR image when
the full 16-bit value is read and interpreted correctly. (This may not be
lossless to half-float pixel values however, which is a very desirable
property for PNG.)
- Some readers may scale the full 16-bit integer to 8-bits, and not just
right shift, so whatever method is used needs to work with them as well.
- An ancillary chunk could hint which global tone mapping operator function
to apply, and which particular parameters, so new PNG software knows what
to do when the caller loads a true HDR image but actually requests a LDR
image.
- An ancillary chunk can also have various other hints, such as the lowest
non-zero value, the largest value of any component, etc. Could also store
the image's average HDR pixel color in log space - all useful for tone
mapping. (Real-time video game engines use similar data to tonemap every
frame.)

---------- Forwarded message ---------
From: Richard Geldreich <rich@binomial.info>
Date: Mon, Nov 6, 2023 at 10:37 PM
Subject: Re: half-float PNG files stored as 16-bit/component images
To: <public-png@w3.org>


Another option is to store the half-float values as 16-bit unsigned but
rotated left one bit, so the half-float sign bit (which is typically 0, and
is stored in the MSB or bit position #15) becomes the LSB (bit position
#0). This is a lossless operation and could be quickly undone by half-float
aware PNG readers. However legacy PNG readers would get a brighter and less
noisy image if they only pay attention to the upper 8-bits. This gets
5-bits of the exponent and 3 bits for the mantissa (so 4 bits total because
the leading '1' bit isn't explicitly stored). I really like this option.

[image: image.png]

On Mon, Nov 6, 2023 at 12:27 PM Richard Geldreich <rich@binomial.info>
wrote:

> Hi all,
> *The one core strength of the PNG image format: predictable backwards
> compatibility.* If we introduce a new Colour Type value (where #7="16-bit
> half float RGBA" for example), *existing PNG decoders will fail to read
> half-float PNG files*. Users will then perceive PNG as an unreliable
> format, which is terrible for a standard file format (if not terminal). It
> could take many years for developers to add proper half-float support, and
> some old libs or software will never be updated. PNG currently works
> everywhere and is very reliable.
>
> *Crucially, Jim Blinn said years ago
> <https://stackoverflow.com/questions/75772363/why-does-the-integer-representation-of-a-floating-point-number-offer-a-piecewise>*
> : "*If you only deal with positive numbers, the bit pattern of a floating
> point number, interpreted as an integer, gives a piecewise linear
> approximation to the logarithm function*".
>
> This holds true for positive half floats interpreted as unsigned 16-bit
> values as well. Half floats
> <https://en.wikipedia.org/wiki/Half-precision_floating-point_format> are
> much like 32-bit floats, just with less mantissa/exponent bits. (This is
> how the BC6H/ASTC HDR GPU texture formats work internally, BTW: the half
> float values are manipulated and interpolated as 16-bit unsigned integers.
> BC6H/ASTC encoders work in half-float space, which is an approximation of a
> logarithmic space.)
>
> One true HDR PNG option is to store the half-float pixel values as plain
> 16-bit/component PNG Truecolour images (Colour Type=2, Bit Depth=16), with
> a special new ancillary chunk that hints to the reader software that it's
> actually a half-float image and not unsigned. *But crucially, what will
> loaded or displayed images look like when processed by legacy PNG software?
> It turns out the results are surprisingly decent.*
>
> Some popular PNG readers (such as stb_image.h, and mine) load
> 16-bit/component images by shifting each component right by 8 bits, to
> return the uppermost 8-bits (so 24 or 32 bits total for RGB/RGBA) to the
> caller. In half float space, this is approximately equivalent to
> floor(log2(value)*scale/256.0). *It's a built-in global tone map
> operator. *The upper 8-bits contain the sign (almost always positive, so
> 0), the 5-bit exponent, and the upper 2-bits of the 10-bit mantissa.
>
> Here's an HDR example: *candle-glass.exr*, with a max float component
> value of 413.25, converted to half float. I then extracted the high 8-bits
> of these half floats to make plain 24-bit images for visualization. Here's
> what the result looks like (and what a 16-bit half float PNG would look
> like loaded with legacy software):
>
> [image: image.png]
>
> After a histogram stretch (using Paint Shop Pro):
> [image: image.png]
>
> Alternatively, after a quick brightness/contrast adjustment:
> [image: image.png]
>
> This is a very simple/crude global tone map operator. The user can still
> load the image with old software, they will just have to adjust the
> brightness/contrast, which is probably fine for a true HDR PNG image loaded
> with legacy software.
>
> Another example (MtTamWest.exr):
>
> [image: image.png]
>
> After quick brightness/contrast adjustment:
>
> [image: image.png]
>