# Off-by-one error in double to long long conversion ?

From: João Eiras <joaoe@opera.com>
Date: Thu, 23 Aug 2012 14:55:58 +0200

Message-ID: <op.wjhvjk0e2q99of@joaoe>
```
Hi !

The number to long long and unsigned long long conversion algorithm seems
to have a off-by-one error [1]

It limits long long to the [-2^53+1,2^53-1] range and unsigned long long
to [0,2^53-1];

If we convert the values

-2^53 - 1 = -9007199254740992 (loses unit precision, throws TypeError
according to the spec)
-2^53     = -9007199254740992 (has unit precision, throws TypeError
according to the spec)
-2^53 + 1 = -9007199254740991 (has unit precision, allowed according to
the spec)

2^53 - 1 = 9007199254740991 (has unit precision, allowed according to the
spec)
2^53     = 9007199254740992 (has unit precision, throws TypeError
according to the spec)
2^53 + 1 = 9007199254740992 (loses unit precision, throws TypeError
according to the spec)

it doesn't make sense to disallow -/+2^53 because they still accurately
represent the integer.

Thank you.

[1] http://dev.w3.org/2006/webapi/WebIDL/#es-long-long
```
Received on Thursday, 23 August 2012 12:56:33 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:37:47 UTC