W3C home > Mailing lists > Public > public-script-coord@w3.org > July to September 2012

Off-by-one error in double to long long conversion ?

From: João Eiras <joaoe@opera.com>
Date: Thu, 23 Aug 2012 14:55:58 +0200
To: "public-script-coord@w3.org" <public-script-coord@w3.org>
Message-ID: <op.wjhvjk0e2q99of@joaoe>

Hi !

The number to long long and unsigned long long conversion algorithm seems  
to have a off-by-one error [1]

It limits long long to the [-2^53+1,2^53-1] range and unsigned long long  
to [0,2^53-1];

If we convert the values

-2^53 - 1 = -9007199254740992 (loses unit precision, throws TypeError  
according to the spec)
-2^53     = -9007199254740992 (has unit precision, throws TypeError  
according to the spec)
-2^53 + 1 = -9007199254740991 (has unit precision, allowed according to  
the spec)

2^53 - 1 = 9007199254740991 (has unit precision, allowed according to the  
spec)
2^53     = 9007199254740992 (has unit precision, throws TypeError  
according to the spec)
2^53 + 1 = 9007199254740992 (loses unit precision, throws TypeError  
according to the spec)

it doesn't make sense to disallow -/+2^53 because they still accurately  
represent the integer.

Thank you.

[1] http://dev.w3.org/2006/webapi/WebIDL/#es-long-long
Received on Thursday, 23 August 2012 12:56:33 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 8 May 2013 19:30:07 UTC