Re: IndexedDB, Blobs and partial Blobs - Large Files

I am aware of [1], and really waiting for this to be available.

So you are suggesting something like {id:file_id, chunk1:chunk1, 
chunk2:chunk2, etc}?

Related to [1] I have tried a "workaround" (not for fun, because I 
needed to test at least with two different browsers): store the chunks 
as ArrayBuffers in an Array {id:file_id, [chunk1, chunk2,... ]}, after 
testing different methods the idea was to new Blob([chunk1, chunk2,... 
]) on query and avoid creating a big ArrayBuffer on update.

Unfortunately, with my configuration, Chrome crashes systematically on 
update for "big" files (tested with 250 MB file and chunks of 2 MB, does 
not seem to be something really enormous).

Then I was thinking to use different keys as you suggest but maybe it's 
not very easy to manipulate and you still have to use an Array to 
concatenate, what's the best method?

Regards,

Aymeric

[1] http://code.google.com/p/chromium/issues/detail?id=108012

Le 02/12/2013 23:38, Joshua Bell a écrit :
> On Mon, Dec 2, 2013 at 9:26 AM, Aymeric Vitte <vitteaymeric@gmail.com 
> <mailto:vitteaymeric@gmail.com>> wrote:
>
>     This is about retrieving a large file with partial data and
>     storing it in an incremental way in indexedDB.
>
> ...
>
>     This seems not efficient at all, was it never discussed the
>     possibility to be able to append data directly in indexedDB?
>
>
> You're correct, IndexedDB doesn't have a notion of updating part of a 
> value, or even querying part of a value (other than via indexes). 
> We've received developer feedback that partial data update and query 
> would both be valuable, but haven't put significant thought into how 
> it would be implemented. Conceivably you could imagine an API for 
> "get" or "put" with an additional keypath into the object. We 
> (Chromium) currently treat the stored value as opaque so we'd need to 
> deserialize/reserialize the entire thing anyway unless we added extra 
> smarts in there, at which point a smart caching layer implemented in 
> JS and tuned for the webapp might be more effective.
>
> Blobs are pesky since they're not mutable. So even with the above 
> hand-waved API you'd still be paying for a fetch/concatenate/store. 
> (FWIW, Chromium's support for Blobs in IndexedDB is still in progress, 
> so this is all in the abstract.)
>
> I think the best advice at the moment for dealing with incremental 
> data in IDB is to store the chunks under separate keys, and 
> concatenate when either all of the data has arrived or lazily on use.
>
>

-- 
Peersm : http://www.peersm.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms