Re: backwards compatible half-float PNG test app source+examples

It's interesting that the sBIT chunk describes a process quite similar to
what we're doing to the integer bits of half-floats, to get the high bytes
in a form that resembles sRGB pixels. The "sample values should be scaled
to the full range of possible values" - which is exactly what the proposed
half float encoding does to the high bytes after the sign bit is rotated
left into the LSB.

Half-floats are a IEEE spec, and very popular on GPU's, so I don't see why
PNG cannot define a chunk and a spec for how the standardized half floats
are mapped.


On Mon, Nov 13, 2023 at 10:18 AM Richard Geldreich <rich@binomial.info>
wrote:

> >If you add the `sRGB` tag, so that it is clearly declared that such is
> the case, then at least you have specified the color appropriately.
>
> Not everyone specifies the color appropriately. In practice, PNG is not
> just a format used for *only* viewing files on a monitor. It's also an
> interchange format used between programs. In common cases the colorspace is
> implied or communicated external to .PNG.
>
> Yet users are still able to preview and view their files using common
> OS's, browsers. It's not ideal but the software/OS/tool ecosystem works
> this way in practice.
>
> On Mon, Nov 13, 2023 at 10:13 AM Leonard Rosenthol <lrosenth@adobe.com>
> wrote:
>
>> If you add the `sRGB` tag, so that it is clearly declared that such is
>> the case, then at least you have specified the color appropriately.
>>
>>
>>
>> And if someone wants to treat the values as something else – such as HDR
>> data – they can do so, out of scope to the spec…which is also fine(-ish).
>>
>>
>>
>> Leonard
>>
>>
>>
>> *From: *Richard Geldreich <rich@binomial.info>
>> *Date: *Monday, November 13, 2023 at 10:10 AM
>> *To: *Leonard Rosenthol <lrosenth@adobe.com>
>> *Cc: *public-png@w3.org <public-png@w3.org>
>> *Subject: *Re: backwards compatible half-float PNG test app
>> source+examples
>>
>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>
>>
>>
>> > BUT if they are “backwards compatible” (as you discuss), then at least
>> for that use case, they still need to be relevant to a color space.  And
>> what is that?
>>
>>
>>
>> If we take an .EXR image downloaded from the web without
>> "chromaticities", "whiteLuminance", or "adoptedNeutral" fields (which are
>> defined as strictly optional in the spec itself, and aren't always used) -
>> what colorspace is the output half-float PNG?
>>
>>
>>
>> The high bytes in the half float PNG (which is what Windows and other
>> views will preview and view 16-bit PNG's as) in the lossless encoding I've
>> proposed will be in sRGB.
>>
>>
>>
>> On Mon, Nov 13, 2023 at 10:00 AM Leonard Rosenthol <lrosenth@adobe.com>
>> wrote:
>>
>> But any pixel value is only useful when considered in a given color space
>> – unless, as Chris noted in his message, that you don’t consider these
>> values for display and only for “data storage”.  And if that is the case,
>> then PNG isn’t the right place for them.   BUT if they are “backwards
>> compatible” (as you discuss), then at least for that use case, they still
>> need to be relevant to a color space.  And what is that?
>>
>>
>>
>> Yes, I agree that EXR is a complex format – been working with that team
>> on adding C2PA support, so I get it.
>>
>>
>>
>> But just because you have a hammer, doesn’t make everything a nail…
>>
>>
>>
>> Leonard
>>
>>
>>
>> *From: *Richard Geldreich <rich@binomial.info>
>> *Date: *Monday, November 13, 2023 at 9:52 AM
>> *To: *Leonard Rosenthol <lrosenth@adobe.com>
>> *Cc: *public-png@w3.org <public-png@w3.org>
>> *Subject: *Re: backwards compatible half-float PNG test app
>> source+examples
>>
>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>
>>
>>
>> >First off, HDR doesn’t mean anything without (at least) a CICP table.
>> So are you adding a standard cICP tag as well to your images?
>>
>> The half float encoding != colorspace. Ward: "Because it [half floats]
>> can represent negative primary values along with positive ones, the OpenEXR
>> format covers the entire visible gamut and a range of about 10.7 orders of
>> magnitude with a relative precision of 0.1%." It can handle any colorspace.
>> Also according to Ward, many .EXR images don't actually encode the
>> *optional* " chromaticities", "whiteLuminance", and "adoptedNeutral"
>> .EXR fields in the file, yet somehow we've survived.
>>
>> >Second, what is the benefit to your published, but proprietary solution
>> over an open specification such as OpenEXR?
>>
>> The Windows OS will not preview or view .EXR files, but it will preview
>> and view half-float encoded PNG files if saved correctly in a lossless
>> manner which we've already solved.
>>
>>
>>
>> "TinyEXR" is 10,000 lines of code, not counting the miniz library for
>> Deflate support (which I wrote BTW). It can only load ~70% of the EXR's in
>> the wild, so as a reliable interchange format it's terrible.OpenEXR itself
>> is too much code and too bloated (dozens of source files).
>>
>>
>>
>> The half-float encoding is an industry standard (IEEE 754-2008), and is
>> widely supported in GPU hardware and in GPU shader languages:
>> https://en.wikipedia.org/wiki/Half-precision_floating-point_format
>>
>>
>>
>>
>>
>> On Mon, Nov 13, 2023 at 9:33 AM Leonard Rosenthol <lrosenth@adobe.com>
>> wrote:
>>
>> Richard – I don’t understand.
>>
>>
>>
>> First off, HDR doesn’t mean anything without (at least) a CICP table.  So
>> are you adding a standard cICP tag as well to your images?  If you are
>> doing that, then at least you’re creating something that can be used in
>> other workflows.  This is especially important as you think about how these
>> images will be composited with other HDR content, but in other formats…for
>> example, consider a web page with both one of your PNGs and a JPEG (using
>> the upcoming standard)…
>>
>>
>>
>> Second, what is the benefit to your published, but proprietary solution
>> over an open specification such as OpenEXR?  Just that it happens to work
>> in PNG instead of some other format?  From a wider industry perspective, I
>> don’t see the benefit and instead lots of downsides to users worldwide who
>> will now be confused why their images don’t work as expected.
>>
>>
>>
>> I’m certainly in favor of moving HDR forward – but IMO, we need to do it
>> consistently as an industry and not a series of “one offs”….
>>
>>
>>
>> Leonard
>>
>>
>>
>> *From: *Richard Geldreich <rich@binomial.info>
>> *Date: *Monday, November 13, 2023 at 9:10 AM
>> *To: *Leonard Rosenthol <lrosenth@adobe.com>
>> *Cc: *public-png@w3.org <public-png@w3.org>
>> *Subject: *Re: backwards compatible half-float PNG test app
>> source+examples
>>
>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>
>>
>>
>> > Before introducing another PNG-specific approach, we should see where
>> the industry is going and see if we can align with them first (which I
>> believe we can/should).
>>
>>
>>
>> We need 16-bit/component half float encoding support (not "color" -
>> that's a different problem) right now for our HDR codec work, so I'm going
>> to open source this couple hundred line HDR PNG solution with a small easy
>> to understand readme doc this week, using a ZLIB+Public Domain license.
>> I'll then work with PNG library authors to get it supported into their libs
>> and modify existing open source viewers to load HDR PNG's.
>>
>>
>>
>> We're already using half-float HDR PNG images for testing and development
>> in our codec work here. It's backwards compatible with all existing readers
>> and OS's, unlike .EXR, so it's a no brainer for us.
>>
>>
>>
>> Best regards,
>>
>> -Rich
>>
>>
>>
>> On Mon, Nov 13, 2023 at 8:35 AM Leonard Rosenthol <lrosenth@adobe.com>
>> wrote:
>>
>> Richard – very interesting work, thanks for sharing!
>>
>>
>>
>> However, it isn’t in line with where the rest of the industry is going
>> with respect to HDR imaging.  Currently ISO TC 42, TC 171, TC 130, JTC 1/SC
>> 29 and the ICC are jointly working on a set of proposals for how to encode
>> various aspects of HDR data – including gain maps – into common formats and
>> technologies (including PNG).  The groups will be meeting in Tokyo in
>> February for a full day working session to continue to move this work
>> forward.  I believe that Chris Lilley will be there (perhaps remotely?) as
>> well as myself.
>>
>>
>>
>> We already have some complex logic in PNG with respect to color and HDR
>> (cICP vs. iCCP vs. gAMA vs sRGB vs ….).  Before introducing another
>> PNG-specific approach, we should see where the industry is going and see if
>> we can align with them first (which I believe we can/should).
>>
>>
>>
>> Leonard
>>
>>
>>
>> *From: *Richard Geldreich <rich@binomial.info>
>> *Date: *Friday, November 10, 2023 at 11:05 PM
>> *To: *public-png@w3.org <public-png@w3.org>
>> *Subject: *Re: backwards compatible half-float PNG test app
>> source+examples
>>
>> *EXTERNAL: Use caution when clicking on links or opening attachments.*
>>
>>
>>
>> Here's how the half float pixel values are losslessly recovered from
>> 16-bit/component HDR PNG images in the "hdrpng" example code. This is the
>> entire decoding procedure.
>>
>>
>>
>> This code extracts the low and high bytes from the stored PNG's 16-bpp
>> pixels, remaps the high byte through the lookup table read from the "hdRa"
>> ancillary chunk (below), shifts right the 16-bit value from [0,15] bits
>> (this is typically 0, but for darker images it may be a small # of bits),
>> and then rotates the sign bit back into the MSB from the LSB of the 16-bit
>> value. This standard half float value can then be processed in an HDR
>> pipeline.
>>
>>
>>
>> The lossless encoder that takes half-floats and outputs 16-bit PNG values
>> is more complex, but not by much. It has to determine the shift amount by
>> examining all the half floats in the image and a high byte histogram, and
>> then compute a 256 entry lookup table using some sort of tone mapping
>> algorithm. The table must be computed in a way that results in no loss.
>>
>>
>>
>>
>>
>> On Fri, Nov 10, 2023 at 4:48 AM Richard Geldreich <rich@binomial.info>
>> wrote:
>>
>> If you want to see these example HDR .PNG's, unpacked to HDR .EXR files,
>> using an in-browser HDR viewer, I've unpacked them (using the example
>> hdrpng tool) to github here:
>>
>> https://github.com/richgel999/png16/tree/main/bin/unpacked
>>
>>
>>
>> The EXR viewer app (it's pretty good - I use it for testing on SDR
>> monitors):
>>
>> https://viewer.openhdr.org/
>>
>>
>>
>>
>>
>> On Fri, Nov 10, 2023 at 4:34 AM Richard Geldreich <rich@binomial.info>
>> wrote:
>>
>> I've found a lossless and trivally invertible transform that takes
>> half-float HDR values (typically read from .HDR or .EXR images) and packs
>> them to 16-bit unsigned pixels that are completely compatible with existing
>> non-HDR aware PNG software. It uses a simple invertible and lossless global
>> tone mapping operator that operates directly on the half-float values. Old
>> readers view and see these PNG's as 16-bit PNG's (so usually 48bpp per
>> pixel for RGB).
>>
>>
>>
>> The high bytes are tone mapped so the 16bpp images appear passable to
>> existing readers. New readers can parse a small ~257 byte ancillary chunk
>> which contains a byte remapping table used to remap the high bytes of the
>> 16-bit components in the PNG image back to half-floats. (It's a little bit
>> more complex than this to deal with signed floats and very low values, but
>> that's the gist of it.) It's lossless for all valid half-float values
>> (normals, denormals, signed). I filter out any NaN's/Inf"s in this test.
>>
>>
>>
>> You can see a bunch of example 48-bpp PNG's packed from .EXR images in
>> this way here:
>>
>> https://github.com/richgel999/png16/tree/main/bin
>>
>>
>>
>> The C++ source to the example "hdrpng" tool is here. The example app
>> excluding image reading/writing is only ~450 lines of code. (It currently
>> compiles with VS 2022 under Windows - I'll add a Linux cmake file next.)
>>
>> https://github.com/richgel999/png16/
>>
>>
>>
>> It uses the popular open source lodepng library, unmodified, to write and
>> read 16-bit PNG files and manipulate the ancillary "hdRa" chunk.
>> hdrpng supports packing and tone mapping .EXR images to .PNG, unpacking HDR
>> .PNG to .EXR, and an .EXR file comparison mode to verify that the half
>> float values can be 100% recovered from the PNG file with no loss. The tone
>> mapping in this example is automatic.
>>
>>
>>
>> I've tested the resulting 48bpp PNG files with pngcheck, Windows
>> Explorer, Chrome, Paint Shop Pro, several tools, and the Windows Photo
>> viewer app. So far, so good - they all look fine.
>>
>>
>>
>> Here are a couple example 48bpp PNG files. (Not sure what gmail will do
>> to them, but this is what they look like.) These are how existing PNG
>> readers view these HDR files. The half-float data is 100% preserved in
>> these .PNG files, so HDR capable viewers are able to retrieve the original
>> half-float pixels and do their own tone mapping or HDR processing.
>>
>>
>>
>>
>>
>> These PNG's validate successfully using pngcheck, because they are
>> completely standard PNG files that any reader can load:
>>
>>
>>
>>