Re: IndexedDB: Binary Keys

On May 21, 2012, at 5:03 PM, Jonas Sicking <jonas@sicking.cc> wrote:

> On Mon, May 21, 2012 at 10:09 AM, Joran Greef <joran@ronomon.com> wrote:
>> IndexedDB supports binary values as per the structured clone algorithm
>> as implemented in Chrome and Firefox.
>> 
>> IndexedDB needs to support binary keys (ArrayBuffer, TypedArrays).
>> 
>> Many popular KV stores accept binary keys (BDB, Tokyo, LevelDB). The
>> Chrome implementation of IDB is already serializing keys to binary.
>> 
>> JS is moving more and more towards binary data across the board
>> (WebSockets, TypedArrays, FileSystemAPI). IDB is not quite there if it
>> does not support binary keys.
>> 
>> Binary keys are more efficient than Base 64 encoded keys, e.g. a 128
>> bit key in base 256 is 16 bytes, but 22 bytes in base 64.
>> 
>> Am working on a production system storing 3 million keys in IndexedDB.
>> In about 6 months it will be storing 60 million keys in IndexedDB.
>> 
>> Without support for binary keys, that's 330mb wasted storage
>> (60,000,000 * (22 - 16)) not to mention the wasted CPU overhead spent
>> Base64 encoding and decoding keys.
> 
> I agree that we should introduce this, but I think it's too late to
> add for version one (which is about to go to last call any day now).
> 
> I'd be happy to add it to version 2 though. However the current
> situation regarding binary data in Javascript is still pretty chaotic.
> See for example the resent change to switch a bunch of APIs over from
> ArrayBuffer to ArrayBufferViews.

Seems to me at this scale (the tens of millions of rows the author is examining,) something like the filesystem API or an otherwise low level solution is more appropriate.
> 

After all, we're still just talking about sqlite.

I agree on revisiting this issue.

Received on Tuesday, 22 May 2012 02:25:48 UTC