Input re: Float16Array + HTML Canvas (Color on the Web CG)

Good morning/evening,

Please find below input from the Color on the Web CG regarding
Float16Array in the context of HTML Canvas. This input was prompted by
the following:

https://github.com/w3c/ColorWeb-CG/issues/87

Best,

-- Color on the Web CG chairs

====================

# CG proposal re: Float16Array + HTML Canvas

The ColorWeb CG is seeking consensus with parties interested in
discussing floating-point backing stores for canvas rendering
contexts.

The CG spec proposal is:
https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md

The primary outstanding issue at present is how to represent values
which are read back from the canvas into typed arrays which can be
manipulated from ECMAScript.

The current spec proposal supports readbacks into both
Uint8ClampedArray and Float32Array via ImageDataColorType.

It has been suggested to also add support for readbacks into
Float16Array (https://github.com/w3c/ColorWeb-CG/issues/87). This
would make the specification dependent on the Float16Array type
currently being defined by the ECMAScript committee (ISO/TC 39).

The CG supports the idea of adding support for Float16Array to
ECMAScript. However, the CG feels it is most appropriate to decouple
the ongoing improvements to ECMAScript from the work on floating-point
canvases. Users of the web platform are expressing the need for
floating-point canvas backing stores now; in Chromium, implementation
is underway in crbug.com/1230619. Moreover, readback and CPU-side
manipulation paths are not high-performance, so the additional memory
bandwidth of using Float32Array for this task will not critically
impact applications.

The ColorWeb CG therefore suggests:

1) Moving ahead with
https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md in its
current form, with unorm8 and float32 as options for readback.

2) In parallel, working on adding Float16Array to ECMAScript, and
prototyping it in browsers.

3) Specifying a "float16" enum in ImageDataColorType once
implementation experience has been achieved with Float16Array, .
Applications can feature-detect its support in browsers by catching
exceptions raised by getImageData or createImageData.

# Background

For context, introducing a Float16Array into the Typed Array hierarchy
was discussed several years ago among members of TC39, as well as the
Khronos Group where Typed Arrays originated. At the time, the costs of
supporting this type natively in ECMAScript engines outweighed the
benefits for the following reasons:

1) For practical purposes, Float16Array could be sufficiently
polyfilled in ECMAScript:

https://github.com/petamoriken/float16

Corner cases existed regarding denormalized values, NaNs, and
infinities - but applications could generally achieve their desired
results, with good performance on all ECMAScript engines, using
existing primitives.

2) Direct CPU support for half-float numbers did not seem to exist. At
the time, C libraries which worked with this data type universally
emulated it, rather than using compiler intrinsics on certain
platforms.

3) A significant amount of work would be required in every ECMAScript
engine to make this typed array type perform well.

4) A significant amount of work would be required to specify the
behavior of this numeric type in ECMAScript.

The landscape has changed in recent years. Per
https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Hardware_support,
ARM and x86 processors either support FP16 natively, or will in the
near future. FP16 formats are increasingly heavily used in neural
network evaluation and image processing, including on the web. The
ECMAScript editors may be able to save spec work by referencing IEEE
specifications (which, to be fair, already existed when Float16Array
was originally considered).

Received on Tuesday, 10 January 2023 22:39:14 UTC