Re: the Khronos "glTF" ISO/IEC standard, PNG's, and half float support

>Do not define new chunks that redefine the meaning of existing chunks or
change the interpretation of an existing standardized chunk, e.g., do not
add a new chunk to say that RGB and alpha values actually mean CMYK.

The 16-bit unsigned data is still 16-bit sRGB for purposes of existing PNG
software.  It is viewable, normal 16-bit sRGB data with an optional alpha
channel for purposes of existing PNG viewers and readers. The half texture
data is explicitly remapped or transformed into two 16-bit encodings and up
to two color spaces (the PNG color space, which we're assuming is sRGB, and
the optional recovered color space) simultaneously, which is a new
approach.

This enables this feature to be 100% interoperable with the existing PNG
spec and software ecosystem. It just happens that this 16-bit sRGB data is
also 100% valid half float data when the high bytes are remapped through a
small 256 entry table (which is common to each RGB channel and global for
the entire PNG image), and the bits shifted/rotated to account for any zero
bits in the exponent field and move the sign bit from the LSB to the MSB.
This half float recovery table is communicated to decoders via the
ancillary "haLf" chunk.

We are not saying this 16-bit data is (for example) only in a single
colorspace or encoding, which is what CMYK would be for example. CMYK
loaded by existing PNG readers would show gibberish, which is not the case
with this half float method as I have demonstrated here already by sending
48bpp PNG's to the Working Group's mailing list, which are viewable using
any existing browser or email client. I have verified that the 48bpp half
float encoded PNG example images (using a production version of the "haLf"
chunk) I sent earlier were officially and properly archived by the W3C's
PNG Working Group list server.

Note a valid haLf chunk could use an identity transform (shift=0, no
remapping using an identity table which doesn't change the high bytes), but
that's not the intention of the "haLf" chunk's spec. However this may be
useful to some users or hardware encoders/SIMD decoders that can't or don't
want to pay the "gather" cost of the remapping operation, or the up front
encoding cost of computing and applying the remapping table. The intention
is for the sRGB image to appear tone mapped in a simple way, by compressing
the *entire* dynamic range of the absolute value of the half input content
losslessly into [0,255] sRGB, via using this table.

Note the recovered half float data may not actually be color pixels at all;
for example the half float data could be a height or 3D vector field, or it
could be a normalized or denormalized half float normal map, etc. This is
why the new colorspace chunk is strictly optional, like it is for other
popular HDR formats such as .EXR and the older .HDR/.PIC (Radiance RGBE)
formats. In many cases the user will transmit the image type (normal map,
albedo map, image map, etc.) via a separate signalling mechanism, a
standard like glTF, a convention completely outside of PNG, or via a naming
convention, etc.

On Tue, Nov 14, 2023 at 10:46 AM Leonard Rosenthol <lrosenth@adobe.com>
wrote:
>
> Richard – thanks for the context of your work with respect to glTF.
Certainly useful to understand.
>
>
>
> Back to your proposed chunks.
>
>
>
> Using a `haLf` chunk (details TBD) to identify that the 16 bit component
values should be treated as half floats, seems like a perfectly reasonable
use for PNG.  As you have noted, it is fully backwards compatible with the
existing ecosystem and (IMO) aligned with Annex B (Guidelines for private
chunks, https://www.w3.org/TR/png-3/#newchunks) of PNG.
>
>
>
> The second one you mentioned, however, seems to be (based on the
description below) to break rule #1 in Annex B:
>
> Do not define new chunks that redefine the meaning of existing chunks or
change the interpretation of an existing standardized chunk, e.g., do not
add a new chunk to say that RGB and alpha values actually mean CMYK.
>
> If I understand your description below, it is doing exactly what it being
recommended against.  Would you agree?
>
>
>
> Leonard
>
>
>
> From: Richard Geldreich <rich@binomial.info>
> Date: Tuesday, November 14, 2023 at 9:20 AM
> To: Portable Network Graphics (PNG) Working Group <public-png@w3.org>
> Cc: NTREVETT@NVIDIA.COM <NTREVETT@NVIDIA.COM>
> Subject: the Khronos "glTF" ISO/IEC standard, PNG's, and half float
support
>
> EXTERNAL: Use caution when clicking on links or opening attachments.
>
>
>
> Note I have CC'd this to Neil Trevett, the Khronos Group President and VP
of Developer Echosystems at NVidia. NVidia is also an official and current
W3C Member.
>
>
>
> The Khronos Group, of which Binomial LLC has been a member for several
years, has created the glTF API-neutral and royalty-free runtime asset
delivery format. glTF bridges the gap between 3D content creation tools and
modern graphics applications by providing an efficient, extensible,
interoperable format for the transmission and loading of 3D content. It's
also an ISO/IEC 12113:2022 International Standard.
>
>
>
> Importantly, the glTF specification explicitly refers to the PNG format
standard by name. PNG textures are very commonly utilized in glTF assets
for the delivery of texture signals for 3D content:
>
> https://en.wikipedia.org/wiki/GlTF
>
>
> Some glTF milestones:
> - At SIGGRAPH 2016, Oculus (Meta) announced their adoption of glTF citing
the similarities to their ovrscene format.
> - In October 2016, Microsoft joined the 3D Formats working group at
Khronos to collaborate on glTF.
> - In February 2020, the Smithsonian Institution launched their Open
Access Initiative, releasing approximately 2.8 million 2D images and 3D
models into the public domain, delivering the 3D models in glTF
> - In July 2022, glTF 2.0 was released as an ISO/IEC 12113:2022
International Standard.
>
>
>
> Binomial has been a member of the glTF Working Group for several years.
We have closely worked with Khronos to standardize our Basis Universal GPU
texture interchange format for usage by glTF, currently for LDR/SDR
signals. The universal GPU texture codecs for 2D HDR half float
texture/image signals are currently in production and coming next.
>
> Importantly, the glTF spec defines several texture image file formats. A
glTF image under section "2.2.3. Technical Terminology" is defined by the
spec as "A two dimensional array of pixels encoded as a standardized
bitstream, such as PNG":
>
>
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#reference-texture
>
>
>
> Also see "2.4. glTF Basics", "2.2.4.1. External Specifications",
"2.2.4.2. Media Type Registrations", etc. where PNG is explicitly
mentioned. It's a common file format utilized by glTF users for the
lossless transmission and delivery of texture signal data.
>
>
>
> Unfortunately, as the .PNG specification doesn't yet support half-float
(IEEE 754-2008 ) HDR texture data (even though the half float format is now
ubiquitous in computer graphics), we're going to extend the PNG file format
using a couple private chunks:
>
> 1. One chunk (tentatively named "haLf") will declare to decoders that
the16-bit FP16/HALF data is losslessly and trivially inevitably remapped,
or tone mapped, inside the PNG file as standard 16-bit sRGB texture data.
This will work with L (Greyscale), LA (Greyscale-Alpha), RGB, or RGBA
textures. Currently the interpretation and encoding of the alpha channel
data isn't impacted, only L or RGB, but this may change to also allow half
float alpha data. Existing PNG software will read and view this half float
data as 16-bit sRGB image/texture data, in a backwards compatible manner.
All encodable 16-bit half values are supported by this chunk: signed,
NaN's, InF's, denormals, or normal half float values.
>
> 2. Another (in progress) optional private ancillary chunk will declare
which colorspace the half texture data is in. This chunk will allow a
lossless conversion from Radiance RGBE HDR/PIC format or .EXR's (optional)
colorspace data. An optional 2nd colorspace is needed because the
tonemapped texture data exists in potentially two colorspaces
simultaneously: sRGB 16-bit for backwards compatibility, and another
colorspace with a half float encoding.
>
>
>
> See the PNG specification under section 12.10.1 "Use of private chunks":
"Encoders MAY use private chunks to carry information that need not be
understood by other applications." As the texture data is delivered as
16-bit sRGB, other existing applications don't need to know it's actually
half float texture data.
>
> Additionally see 5.7.3 Defining private chunks: "An ancillary chunk type,
not a critical chunk type, SHOULD be used for all private chunks that store
information that is not absolutely essential to view the image." These
chunks are not essential to view the image, as the texture data has been
remapped or losslessly and inevitably tone mapped in a backwards compatible
manner. Windows, Chrome, Firefox, and existing PNG image editors all
preview, display and load the 16-bit image data in the usual way, as sRGB
content.
>
>
>
> I'll be writing the half float chunk definition specs and PNG ecosystem
software that supports these chunks over the upcoming months, as part of
our HDR Universal Texture work. We and our customers also use the .PNG file
format to feed LDR/SDR texture signal data into our Universal GPU texture
codec, and once this addition is made they'll also be able to input into
our codec half float texture data. Once we roll this out glTF standard
users will be able to feed in losslessly compressed HDR (via half float
PNG) and lossy HDR texture/image signal data (via Basis Universal HDR,
which will support half float PNG) into glTF, optionally in any colorspace.
>
>
>
> Best regards,
>
> -Rich
>
>
>
>

>