IndexedDB, what were the issues? How do we stop it from happening again?

Comments inline. Adding some folks from the IDB team at Google to the
thread as well as public-webapps.

On Sunday, February 17, 2013, Miko Nieminen wrote:

>
>
> 2013/2/15 Shwetank Dixit <shwetankd@opera.com>
>
>>  Why did you feel it was necessary to write a layer on top of IndexedDB?
>>>
>>
>> I think this is the main issue here.
>>
>> As it stands, IDB is great in terms of features and power it offers, but
>> the feedback I recieved from other devs was that writing raw IndexedDB
>> requires an uncomfortable amount of verbosity even for some simple tasks
>> (This can be disputed, but that is the views I got from some of the
>> developers I interacted with). Adding that much amount of code (once again,
>> im talking of raw IndexedDB) makes it less readable and understandable. For
>> beginners, this all seemed very intimidating, and for some people more
>> experienced, it was a bit frustrating.
>>
>>
> After my experiments with IDB, I don't feel that it is particularly
> verbose. I have to admit that often I prefer slightly verbose syntax over
> shorter one when it makes reading the code easier. In IDB's case, I think
> this is the case.
>
>
>
>>  For the latter bit, I reckon it would be a good practice for groups
>>> working on low-level APIs to more or less systematically produce a library
>>> that operates at a higher level. This would not only help developers in
>>> that they could pick that up instead of the lower-level stuff, but more
>>> importantly (at least in terms of goals) it would serve to validate that
>>> the lower-level design is indeed appropriate for librarification.
>>>
>>
>> I think that would be a good idea. Also, people making those low level
>> APIs should still keep in mind that the resulting code should not be too
>> verbose or complex. Librarification should be an advantage, but not a de
>> facto requirement for developers when it comes to such APIs. It should
>> still be feasable for them to write code in the raw low level API without
>> writing uncomfortably verbose or complex code for simple tasks. Spec
>> designers of low level APIs should not take this as a license to make
>> things so complex that only they and a few others understand it, and then
>> hope that some others will go ahead and make it simple for the 'common
>> folk' through an abstraction library.
>
>
> I quite don't see how to simplify IDB syntax much more.
>

I've avoided weighing in on this thread until I had more IDB experience.
I've been wrestling with it on two fronts of late:


   - A re-interpretation of the API based on Futures:

   https://github.com/slightlyoff/DOMFuture/tree/master/reworked_APIs/IndexedDB
   - A new async LocalStorage design + p(r)olyfill that's bootstrapped on
   IDB:
   https://github.com/slightlyoff/async-local-storage

While you might be right that it's unlikely that the API can be
"simplified", I think it's trivial to extend it in ways that make it easier
to reason about and use.

This thread started out with a discussion of what might be done to keep
IDB's perceived mistakes from reoccurring. Here's a quick stab at both an
outline of the mistakes and what can be done to avoid them:


   - *Abuse of events*
   The current IDB design models one-time operations using events. This *can
   * make sense insofar as events can occur zero or more times in the
   future, but it's not a natural fit. What does it mean for oncomplete to
   happen more than once? Is that an error? Are onsuccess and onerror
   exclusive? Can they both be dispatched for an operation? The API isn't
   clear. Events don't lead to good design here as they don't encapsulate
   these concerns. Similarly, event handlers don't chain. This is natural, as
   they could be invoked multiple times (conceptually), but it's not a good
   fit for data access. It's great that IDB as async, and events are the
   existing DOM model for this, but IDB's IDBRequest object is calling out for
   a different kind of abstraction. I'll submit Futures for the job, but
   others might work (explicit callback, whatever) so long as they maintain
   chainability + async.

   - *Implicitness*
   IDB is implicit in a number of places that cause confusion for folks
   not intimately familiar with the contract(s) that IDB expects you to enter
   into. First, the use of events for delivery of notifications means that
   sequential-looking code that you might expect to have timing issues
   doesn't. Why not? Because IDB operates in some vaguely async way; you can't
   reason at all about events that have occurred in the past (they're not
   values, they're points in time). I can't find anywhere in the spec that the
   explicit gaurantees about delivery timing are noted (
   http://www.w3.org/TR/IndexedDB/#async-api), so one could read IDB code
   that registers two callbacks as having a temporal dead-zone: a space in
   code where something might have happened but which your code might not have
   a chance to hear about. I realize that in practice this isn't the case;
   event delivery for these is asynchronous, but the soonest timing isn't
   defined: end of turn? next turn? end-of-microtask? This means that it's
   possible to have implementations the differ on delivery timing, astonishing
   those who register event handlers at the wrong time. This is part DOM-ish
   use of events for things they're not suited to and a lack of specificity in
   the spec. Both can be fixed.

   A related bit of implicitness is the transaction object. Auto-open and
   auto-close might be virtues, but they come with costs. *When* does a
   transaction auto-close? It's not clear from the spec; 4.2 says that a
   transaction must be inactive when control returns to the event loop, but
   gives no indication of what the nearest timing for that is. It's also not
   clear how to keep a transaction "alive" across turns (a basic need), create
   sub-transactions (a key feature of many transaction-oriented DBs), and
   detect that a transaction object is in something other than the "active"
   state. The last bit is particularly galling: you can have a handle to the
   object, but users can't ask for state they might want, despite the spec
   spending a great deal of time telling implementers that they must do this
   and that with this bit. If there's a principle at issue, it's the idea that
   specs -- particularly low-level APIs -- should not reserve to themselves
   state and information that they need but for which they don't immediately
   spot a user need. There's an obvious exception in the case of security
   boundaries, but that's a different thing entirely. Generally speaking, if
   you need it when writing down how your API operates, your users will too.
   It's particularly punitive to be throwing exceptions for violations of
   state you can't inspect but could manually cobble together from a large set
   of events.

   - *Confused collection interfaces
   *IDB has a factory for databases and object stores and
   allows retrieval of them by name (asynchronously, which is good)...but
   doesn't provide a coherent Map interface onto them. By being DOM-ish and
   not JS-ish, IDB once again creates oddball JS objects that could pun with
   built-ins and therefore ease the learning curve, but doesn't. No, these
   aren't (synchronous) maps, but punning the API with ES6's Map type would go
   a long way.

   - *Doubled API surface for sync version*
   I assume I just don't understand why this choice was made, but the
   explosion of API surface area combined with the conditional availability of
   this version of the API make it an odd beast (to be charitable).

   - *The idea that this is all going to be wrapped up by libraries anyway*
   This is aesthetic, as therefore subjective, but IDB is not a beautiful
   API; nor does it seem evident that beauty and clarity were explicit goals.
   I wasn't involved and don't know all the motivations (nor do I have time to
   read all the minutes now), but there seems to be some apology happening now
   for the lack of beauty and usability the the API based on the idea that
   it'll just be wrapped up by libraries. This is failure for an API designer;
   we should recognize it as such and try to belay it as long as possible.
   Yes, all APIs are eventually wrapped as our general level of abstraction
   goes up the stack, but it's possible to provide solid, usable APIs that
   stand the test of time. Certainly no *new* API should plan on being as
   painful to use as DOM has been historically.

I'll close by saying that all of this is tractable. We can retrofit
IDBRequest to be a Future subclass, create a Map-alike interface for the
list of DB's and object stores, and move away from events where they're not
natural; all without breaking the API. I'm hopeful we can do it quickly.


> I think its request object based API is very nice and transactions are
> much appreciated. Possible simplification could be achieved by introducing
> somekind of auto transaction mechanism so that user could get and change
> objects without creating transactions. There are some challenges to enable
> this and it would complicate the engine especially if transactions are
> still supported when users want to use those. And I hope transactions are
> not dropped completely. When using CouchDB, I often find my self writing
> some fairly painful code to handle the lack of transactions.
>
> Since IDB is aiming for its first standardised version of the API, I
> wouldn't be too worried about people writing Javascript libraries that
> simplify its use. As long as all low level capabilities are in place for
> writing these abstractions, we should be in good order for the first
> version of the standard. Later in following versions of the API we have
> more experience about painful parts of IDB API and we can improve it and
> simplify its use. Extending API by creating additional abstractions to
> simplify its use is often more easier than going to other direction at
> least according to my experience.
>
> --
> Miko Nieminen
> miko.nieminen@iki.fi
> miko.nieminen@gmail.com
>
>