Re: the Khronos "glTF" ISO/IEC standard, PNG's, and half float support

Richard – thanks for the context of your work with respect to glTF.  Certainly useful to understand.

Back to your proposed chunks.

Using a `haLf` chunk (details TBD) to identify that the 16 bit component values should be treated as half floats, seems like a perfectly reasonable use for PNG.  As you have noted, it is fully backwards compatible with the existing ecosystem and (IMO) aligned with Annex B (Guidelines for private chunks, https://www.w3.org/TR/png-3/#newchunks) of PNG.

The second one you mentioned, however, seems to be (based on the description below) to break rule #1 in Annex B:

  1.  Do not define new chunks that redefine the meaning of existing chunks or change the interpretation of an existing standardized chunk, e.g., do not add a new chunk to say that RGB and alpha values actually mean CMYK.
If I understand your description below, it is doing exactly what it being recommended against.  Would you agree?

Leonard

From: Richard Geldreich <rich@binomial.info>
Date: Tuesday, November 14, 2023 at 9:20 AM
To: Portable Network Graphics (PNG) Working Group <public-png@w3.org>
Cc: NTREVETT@NVIDIA.COM <NTREVETT@NVIDIA.COM>
Subject: the Khronos "glTF" ISO/IEC standard, PNG's, and half float support

EXTERNAL: Use caution when clicking on links or opening attachments.


Note I have CC'd this to Neil Trevett, the Khronos Group President and VP of Developer Echosystems at NVidia. NVidia is also an official and current W3C Member.

The Khronos Group, of which Binomial LLC has been a member for several years, has created the glTF API-neutral and royalty-free runtime asset delivery format. glTF bridges the gap between 3D content creation tools and modern graphics applications by providing an efficient, extensible, interoperable format for the transmission and loading of 3D content. It's also an ISO/IEC 12113:2022 International Standard.

Importantly, the glTF specification explicitly refers to the PNG format standard by name. PNG textures are very commonly utilized in glTF assets for the delivery of texture signals for 3D content:
https://en.wikipedia.org/wiki/GlTF

Some glTF milestones:
- At SIGGRAPH 2016, Oculus (Meta) announced their adoption of glTF citing the similarities to their ovrscene format.
- In October 2016, Microsoft joined the 3D Formats working group at Khronos to collaborate on glTF.
- In February 2020, the Smithsonian Institution launched their Open Access Initiative, releasing approximately 2.8 million 2D images and 3D models into the public domain, delivering the 3D models in glTF
- In July 2022, glTF 2.0 was released as an ISO/IEC 12113:2022 International Standard.

Binomial has been a member of the glTF Working Group for several years. We have closely worked with Khronos to standardize our Basis Universal GPU texture interchange format for usage by glTF, currently for LDR/SDR signals. The universal GPU texture codecs for 2D HDR half float texture/image signals are currently in production and coming next.

Importantly, the glTF spec defines several texture image file formats. A glTF image under section "2.2.3. Technical Terminology" is defined by the spec as "A two dimensional array of pixels encoded as a standardized bitstream, such as PNG":
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#reference-texture

Also see "2.4. glTF Basics", "2.2.4.1. External Specifications", "2.2.4.2. Media Type Registrations", etc. where PNG is explicitly mentioned. It's a common file format utilized by glTF users for the lossless transmission and delivery of texture signal data.

Unfortunately, as the .PNG specification doesn't yet support half-float (IEEE 754-2008 ) HDR texture data (even though the half float format is now ubiquitous in computer graphics), we're going to extend the PNG file format using a couple private chunks:
1. One chunk (tentatively named "haLf") will declare to decoders that the16-bit FP16/HALF data is losslessly and trivially inevitably remapped, or tone mapped, inside the PNG file as standard 16-bit sRGB texture data. This will work with L (Greyscale), LA (Greyscale-Alpha), RGB, or RGBA textures. Currently the interpretation and encoding of the alpha channel data isn't impacted, only L or RGB, but this may change to also allow half float alpha data. Existing PNG software will read and view this half float data as 16-bit sRGB image/texture data, in a backwards compatible manner. All encodable 16-bit half values are supported by this chunk: signed, NaN's, InF's, denormals, or normal half float values.
2. Another (in progress) optional private ancillary chunk will declare which colorspace the half texture data is in. This chunk will allow a lossless conversion from Radiance RGBE HDR/PIC format or .EXR's (optional) colorspace data. An optional 2nd colorspace is needed because the tonemapped texture data exists in potentially two colorspaces simultaneously: sRGB 16-bit for backwards compatibility, and another colorspace with a half float encoding.

See the PNG specification under section 12.10.1 "Use of private chunks": "Encoders MAY use private chunks to carry information that need not be understood by other applications." As the texture data is delivered as 16-bit sRGB, other existing applications don't need to know it's actually half float texture data.
Additionally see 5.7.3 Defining private chunks: "An ancillary chunk type, not a critical chunk type, SHOULD be used for all private chunks that store information that is not absolutely essential to view the image." These chunks are not essential to view the image, as the texture data has been remapped or losslessly and inevitably tone mapped in a backwards compatible manner. Windows, Chrome, Firefox, and existing PNG image editors all preview, display and load the 16-bit image data in the usual way, as sRGB content.

I'll be writing the half float chunk definition specs and PNG ecosystem software that supports these chunks over the upcoming months, as part of our HDR Universal Texture work. We and our customers also use the .PNG file format to feed LDR/SDR texture signal data into our Universal GPU texture codec, and once this addition is made they'll also be able to input into our codec half float texture data. Once we roll this out glTF standard users will be able to feed in losslessly compressed HDR (via half float PNG) and lossy HDR texture/image signal data (via Basis Universal HDR, which will support half float PNG) into glTF, optionally in any colorspace.

Best regards,
-Rich