Re: backwards compatible half-float PNG test app source+examples

After thinking about it, if the PNG spec just defined how to "recover" or
decode standard 16-bit/component PNG's (in say the sRGB colorspace) to IEEE
half-floats, in a sBIT-style chunk, that's all we need to get going in a
practical way.

The specific encoding/tone mapping method can be left undefined in
the spec, because developers will come up with creative ways of solving
that problem. The sBIT-like chunk can include a 256 entry remapping table
and a bit shift amount. All the spec needs to define is the half float
recovery process.

An optional way of communicating "chromaticities", "whiteLuminance", and
"adoptedNeutral" fields of the recovered half float data, in a way that can
be transcoded losslessly to/from .EXR, would then be needed as well. The
PNG would exist in two encodings or colorspaces simultaneously.

On Mon, Nov 13, 2023 at 10:34 AM Richard Geldreich <rich@binomial.info>
wrote:

> Note the proposed half float encoding exists in *two colorspaces (or
> encodings) simultaneously *because it's been tone mapped: 16-bits sRGB
> and another space with a half float encoding, which may be undefined in the
> file.
>
> Because it's in 16-bit sRGB it's backwards compatible with virtually all
> existing viewers/readers. That's a very powerful property.
>
>
> On Mon, Nov 13, 2023 at 10:25 AM Richard Geldreich <rich@binomial.info>
> wrote:
>
>> It's interesting that the sBIT chunk describes a process quite similar to
>> what we're doing to the integer bits of half-floats, to get the high bytes
>> in a form that resembles sRGB pixels. The "sample values should be
>> scaled to the full range of possible values" - which is exactly what the
>> proposed half float encoding does to the high bytes after the sign bit is
>> rotated left into the LSB.
>>
>> Half-floats are a IEEE spec, and very popular on GPU's, so I don't see
>> why PNG cannot define a chunk and a spec for how the standardized half
>> floats are mapped.
>>
>>
>> On Mon, Nov 13, 2023 at 10:18 AM Richard Geldreich <rich@binomial.info>
>> wrote:
>>
>>> >If you add the `sRGB` tag, so that it is clearly declared that such is
>>> the case, then at least you have specified the color appropriately.
>>>
>>> Not everyone specifies the color appropriately. In practice, PNG is not
>>> just a format used for *only* viewing files on a monitor. It's also an
>>> interchange format used between programs. In common cases the colorspace is
>>> implied or communicated external to .PNG.
>>>
>>> Yet users are still able to preview and view their files using common
>>> OS's, browsers. It's not ideal but the software/OS/tool ecosystem works
>>> this way in practice.
>>>
>>> On Mon, Nov 13, 2023 at 10:13 AM Leonard Rosenthol <lrosenth@adobe.com>
>>> wrote:
>>>
>>>> If you add the `sRGB` tag, so that it is clearly declared that such is
>>>> the case, then at least you have specified the color appropriately.
>>>>
>>>>
>>>>
>>>> And if someone wants to treat the values as something else – such as
>>>> HDR data – they can do so, out of scope to the spec…which is also
>>>> fine(-ish).
>>>>
>>>>
>>>>
>>>> Leonard
>>>>
>>>>
>>>>
>>>> *From: *Richard Geldreich <rich@binomial.info>
>>>> *Date: *Monday, November 13, 2023 at 10:10 AM
>>>> *To: *Leonard Rosenthol <lrosenth@adobe.com>
>>>> *Cc: *public-png@w3.org <public-png@w3.org>
>>>> *Subject: *Re: backwards compatible half-float PNG test app
>>>> source+examples
>>>>
>>>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>>>
>>>>
>>>>
>>>> > BUT if they are “backwards compatible” (as you discuss), then at
>>>> least for that use case, they still need to be relevant to a color space.
>>>> And what is that?
>>>>
>>>>
>>>>
>>>> If we take an .EXR image downloaded from the web without
>>>> "chromaticities", "whiteLuminance", or "adoptedNeutral" fields (which are
>>>> defined as strictly optional in the spec itself, and aren't always used) -
>>>> what colorspace is the output half-float PNG?
>>>>
>>>>
>>>>
>>>> The high bytes in the half float PNG (which is what Windows and other
>>>> views will preview and view 16-bit PNG's as) in the lossless encoding I've
>>>> proposed will be in sRGB.
>>>>
>>>>
>>>>
>>>> On Mon, Nov 13, 2023 at 10:00 AM Leonard Rosenthol <lrosenth@adobe.com>
>>>> wrote:
>>>>
>>>> But any pixel value is only useful when considered in a given color
>>>> space – unless, as Chris noted in his message, that you don’t consider
>>>> these values for display and only for “data storage”.  And if that is the
>>>> case, then PNG isn’t the right place for them.   BUT if they are “backwards
>>>> compatible” (as you discuss), then at least for that use case, they still
>>>> need to be relevant to a color space.  And what is that?
>>>>
>>>>
>>>>
>>>> Yes, I agree that EXR is a complex format – been working with that team
>>>> on adding C2PA support, so I get it.
>>>>
>>>>
>>>>
>>>> But just because you have a hammer, doesn’t make everything a nail…
>>>>
>>>>
>>>>
>>>> Leonard
>>>>
>>>>
>>>>
>>>> *From: *Richard Geldreich <rich@binomial.info>
>>>> *Date: *Monday, November 13, 2023 at 9:52 AM
>>>> *To: *Leonard Rosenthol <lrosenth@adobe.com>
>>>> *Cc: *public-png@w3.org <public-png@w3.org>
>>>> *Subject: *Re: backwards compatible half-float PNG test app
>>>> source+examples
>>>>
>>>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>>>
>>>>
>>>>
>>>> >First off, HDR doesn’t mean anything without (at least) a CICP table.
>>>> So are you adding a standard cICP tag as well to your images?
>>>>
>>>> The half float encoding != colorspace. Ward: "Because it [half floats]
>>>> can represent negative primary values along with positive ones, the OpenEXR
>>>> format covers the entire visible gamut and a range of about 10.7 orders of
>>>> magnitude with a relative precision of 0.1%." It can handle any colorspace.
>>>> Also according to Ward, many .EXR images don't actually encode the
>>>> *optional* " chromaticities", "whiteLuminance", and "adoptedNeutral"
>>>> .EXR fields in the file, yet somehow we've survived.
>>>>
>>>> >Second, what is the benefit to your published, but proprietary
>>>> solution over an open specification such as OpenEXR?
>>>>
>>>> The Windows OS will not preview or view .EXR files, but it will preview
>>>> and view half-float encoded PNG files if saved correctly in a lossless
>>>> manner which we've already solved.
>>>>
>>>>
>>>>
>>>> "TinyEXR" is 10,000 lines of code, not counting the miniz library for
>>>> Deflate support (which I wrote BTW). It can only load ~70% of the EXR's in
>>>> the wild, so as a reliable interchange format it's terrible.OpenEXR itself
>>>> is too much code and too bloated (dozens of source files).
>>>>
>>>>
>>>>
>>>> The half-float encoding is an industry standard (IEEE 754-2008), and is
>>>> widely supported in GPU hardware and in GPU shader languages:
>>>> https://en.wikipedia.org/wiki/Half-precision_floating-point_format
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Nov 13, 2023 at 9:33 AM Leonard Rosenthol <lrosenth@adobe.com>
>>>> wrote:
>>>>
>>>> Richard – I don’t understand.
>>>>
>>>>
>>>>
>>>> First off, HDR doesn’t mean anything without (at least) a CICP table.
>>>> So are you adding a standard cICP tag as well to your images?  If you are
>>>> doing that, then at least you’re creating something that can be used in
>>>> other workflows.  This is especially important as you think about how these
>>>> images will be composited with other HDR content, but in other formats…for
>>>> example, consider a web page with both one of your PNGs and a JPEG (using
>>>> the upcoming standard)…
>>>>
>>>>
>>>>
>>>> Second, what is the benefit to your published, but proprietary solution
>>>> over an open specification such as OpenEXR?  Just that it happens to work
>>>> in PNG instead of some other format?  From a wider industry perspective, I
>>>> don’t see the benefit and instead lots of downsides to users worldwide who
>>>> will now be confused why their images don’t work as expected.
>>>>
>>>>
>>>>
>>>> I’m certainly in favor of moving HDR forward – but IMO, we need to do
>>>> it consistently as an industry and not a series of “one offs”….
>>>>
>>>>
>>>>
>>>> Leonard
>>>>
>>>>
>>>>
>>>> *From: *Richard Geldreich <rich@binomial.info>
>>>> *Date: *Monday, November 13, 2023 at 9:10 AM
>>>> *To: *Leonard Rosenthol <lrosenth@adobe.com>
>>>> *Cc: *public-png@w3.org <public-png@w3.org>
>>>> *Subject: *Re: backwards compatible half-float PNG test app
>>>> source+examples
>>>>
>>>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>>>
>>>>
>>>>
>>>> > Before introducing another PNG-specific approach, we should see where
>>>> the industry is going and see if we can align with them first (which I
>>>> believe we can/should).
>>>>
>>>>
>>>>
>>>> We need 16-bit/component half float encoding support (not "color" -
>>>> that's a different problem) right now for our HDR codec work, so I'm going
>>>> to open source this couple hundred line HDR PNG solution with a small easy
>>>> to understand readme doc this week, using a ZLIB+Public Domain license.
>>>> I'll then work with PNG library authors to get it supported into their libs
>>>> and modify existing open source viewers to load HDR PNG's.
>>>>
>>>>
>>>>
>>>> We're already using half-float HDR PNG images for testing and
>>>> development in our codec work here. It's backwards compatible with all
>>>> existing readers and OS's, unlike .EXR, so it's a no brainer for us.
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> -Rich
>>>>
>>>>
>>>>
>>>> On Mon, Nov 13, 2023 at 8:35 AM Leonard Rosenthol <lrosenth@adobe.com>
>>>> wrote:
>>>>
>>>> Richard – very interesting work, thanks for sharing!
>>>>
>>>>
>>>>
>>>> However, it isn’t in line with where the rest of the industry is going
>>>> with respect to HDR imaging.  Currently ISO TC 42, TC 171, TC 130, JTC 1/SC
>>>> 29 and the ICC are jointly working on a set of proposals for how to encode
>>>> various aspects of HDR data – including gain maps – into common formats and
>>>> technologies (including PNG).  The groups will be meeting in Tokyo in
>>>> February for a full day working session to continue to move this work
>>>> forward.  I believe that Chris Lilley will be there (perhaps remotely?) as
>>>> well as myself.
>>>>
>>>>
>>>>
>>>> We already have some complex logic in PNG with respect to color and HDR
>>>> (cICP vs. iCCP vs. gAMA vs sRGB vs ….).  Before introducing another
>>>> PNG-specific approach, we should see where the industry is going and see if
>>>> we can align with them first (which I believe we can/should).
>>>>
>>>>
>>>>
>>>> Leonard
>>>>
>>>>
>>>>
>>>> *From: *Richard Geldreich <rich@binomial.info>
>>>> *Date: *Friday, November 10, 2023 at 11:05 PM
>>>> *To: *public-png@w3.org <public-png@w3.org>
>>>> *Subject: *Re: backwards compatible half-float PNG test app
>>>> source+examples
>>>>
>>>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>>>
>>>>
>>>>
>>>> Here's how the half float pixel values are losslessly recovered from
>>>> 16-bit/component HDR PNG images in the "hdrpng" example code. This is the
>>>> entire decoding procedure.
>>>>
>>>>
>>>>
>>>> This code extracts the low and high bytes from the stored PNG's 16-bpp
>>>> pixels, remaps the high byte through the lookup table read from the "hdRa"
>>>> ancillary chunk (below), shifts right the 16-bit value from [0,15] bits
>>>> (this is typically 0, but for darker images it may be a small # of bits),
>>>> and then rotates the sign bit back into the MSB from the LSB of the 16-bit
>>>> value. This standard half float value can then be processed in an HDR
>>>> pipeline.
>>>>
>>>>
>>>>
>>>> The lossless encoder that takes half-floats and outputs 16-bit PNG
>>>> values is more complex, but not by much. It has to determine the shift
>>>> amount by examining all the half floats in the image and a high byte
>>>> histogram, and then compute a 256 entry lookup table using some sort of
>>>> tone mapping algorithm. The table must be computed in a way that results in
>>>> no loss.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Nov 10, 2023 at 4:48 AM Richard Geldreich <rich@binomial.info>
>>>> wrote:
>>>>
>>>> If you want to see these example HDR .PNG's, unpacked to HDR .EXR
>>>> files, using an in-browser HDR viewer, I've unpacked them (using the
>>>> example hdrpng tool) to github here:
>>>>
>>>> https://github.com/richgel999/png16/tree/main/bin/unpacked
>>>>
>>>>
>>>>
>>>> The EXR viewer app (it's pretty good - I use it for testing on SDR
>>>> monitors):
>>>>
>>>> https://viewer.openhdr.org/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Nov 10, 2023 at 4:34 AM Richard Geldreich <rich@binomial.info>
>>>> wrote:
>>>>
>>>> I've found a lossless and trivally invertible transform that takes
>>>> half-float HDR values (typically read from .HDR or .EXR images) and packs
>>>> them to 16-bit unsigned pixels that are completely compatible with existing
>>>> non-HDR aware PNG software. It uses a simple invertible and lossless global
>>>> tone mapping operator that operates directly on the half-float values. Old
>>>> readers view and see these PNG's as 16-bit PNG's (so usually 48bpp per
>>>> pixel for RGB).
>>>>
>>>>
>>>>
>>>> The high bytes are tone mapped so the 16bpp images appear passable to
>>>> existing readers. New readers can parse a small ~257 byte ancillary chunk
>>>> which contains a byte remapping table used to remap the high bytes of the
>>>> 16-bit components in the PNG image back to half-floats. (It's a little bit
>>>> more complex than this to deal with signed floats and very low values, but
>>>> that's the gist of it.) It's lossless for all valid half-float values
>>>> (normals, denormals, signed). I filter out any NaN's/Inf"s in this test.
>>>>
>>>>
>>>>
>>>> You can see a bunch of example 48-bpp PNG's packed from .EXR images in
>>>> this way here:
>>>>
>>>> https://github.com/richgel999/png16/tree/main/bin
>>>>
>>>>
>>>>
>>>> The C++ source to the example "hdrpng" tool is here. The example app
>>>> excluding image reading/writing is only ~450 lines of code. (It currently
>>>> compiles with VS 2022 under Windows - I'll add a Linux cmake file next.)
>>>>
>>>> https://github.com/richgel999/png16/
>>>>
>>>>
>>>>
>>>> It uses the popular open source lodepng library, unmodified, to write
>>>> and read 16-bit PNG files and manipulate the ancillary "hdRa" chunk.
>>>> hdrpng supports packing and tone mapping .EXR images to .PNG, unpacking HDR
>>>> .PNG to .EXR, and an .EXR file comparison mode to verify that the half
>>>> float values can be 100% recovered from the PNG file with no loss. The tone
>>>> mapping in this example is automatic.
>>>>
>>>>
>>>>
>>>> I've tested the resulting 48bpp PNG files with pngcheck, Windows
>>>> Explorer, Chrome, Paint Shop Pro, several tools, and the Windows Photo
>>>> viewer app. So far, so good - they all look fine.
>>>>
>>>>
>>>>
>>>> Here are a couple example 48bpp PNG files. (Not sure what gmail will do
>>>> to them, but this is what they look like.) These are how existing PNG
>>>> readers view these HDR files. The half-float data is 100% preserved in
>>>> these .PNG files, so HDR capable viewers are able to retrieve the original
>>>> half-float pixels and do their own tone mapping or HDR processing.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> These PNG's validate successfully using pngcheck, because they are
>>>> completely standard PNG files that any reader can load:
>>>>
>>>>
>>>>
>>>>