Re: [fxtf-drafts] [filter-effects-1] inconsistent number of decimal places for hueRotate matrix (#542)

In general the level of imprecision reflects the age of the specification, and there is little attempt to minimize the effects of cumulative round-off error.

> It’s also not luminance but "luma", but that’s wayyyy too late. A note might be added to appease pedants like me though.

Yup, all these calculations are being done on gamma-encoded values instead of in the linear-light domain, which is incorrect, but changing that would alter existing content in very visible ways. I agree that this should at least be noted in the specification.

Given that, there is limited utility in increasing the precision of the constants used in these calculations.

However, if there is interest in doing so, [CSS Color 4](https://drafts.csswg.org/css-color-4/#color-conversion-code) defines the conversion in terms of rational numbers, so (looking at the second row of the matrix in `lin_sRGB_to_XYZ`:

```js
function lin_sRGB_to_XYZ(rgb) {
 // convert an array of linear-light sRGB values to CIE XYZ
 // using sRGB's own white, D65 (no chromatic adaptation)

 var M = [
  [ 506752 / 1228815,  87881 / 245763,   12673 /   70218 ],
  [  87098 /  409605, 175762 / 245763,   12673 /  175545 ],
  [   7918 /  409605,  87881 / 737289, 1001167 / 1053270 ],
 ];
 return multiplyMatrices(M, rgb);
}
```

and thus:

- 87,098 ÷ 409,605 = 0.21263900587
- 175,762 ÷ 245,763 = 0.71516867877
- 12,673 ÷ 175,545 = 0.07219231536

so it would be easily possible to round these to a consistent 4 significant figures, at least.

-- 
GitHub Notification of comment by svgeesus
Please view or discuss this issue at https://github.com/w3c/fxtf-drafts/issues/542#issuecomment-1880974113 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 8 January 2024 13:07:31 UTC