Re: IndexedDB, what were the issues? How do we stop it from happening again?

I wrote a quick overview of the issues I have had using the indexedDB API

http://arandomurl.com/2013/02/21/thoughts-on-indexeddb.html

Most of them are just implementation details, however I still havent met a
webdev who understands the transaction model without having one of the
indexeddb implementors explain it to them, and past being confusing it also
makes things much harder to implement due to being tied to the event loop.

and I agree with pretty much everything that Alex wrote, at least the parts
that I understood.

Cheers
Dale

On 6 March 2013 15:01, Alex Russell <slightlyoff@google.com> wrote:

> Comments inline. Adding some folks from the IDB team at Google to the
> thread as well as public-webapps.
>
> On Sunday, February 17, 2013, Miko Nieminen wrote:
>
>>
>>
>> 2013/2/15 Shwetank Dixit <shwetankd@opera.com>
>>
>>>  Why did you feel it was necessary to write a layer on top of IndexedDB?
>>>>
>>>
>>> I think this is the main issue here.
>>>
>>> As it stands, IDB is great in terms of features and power it offers, but
>>> the feedback I recieved from other devs was that writing raw IndexedDB
>>> requires an uncomfortable amount of verbosity even for some simple tasks
>>> (This can be disputed, but that is the views I got from some of the
>>> developers I interacted with). Adding that much amount of code (once again,
>>> im talking of raw IndexedDB) makes it less readable and understandable. For
>>> beginners, this all seemed very intimidating, and for some people more
>>> experienced, it was a bit frustrating.
>>>
>>>
>> After my experiments with IDB, I don't feel that it is particularly
>> verbose. I have to admit that often I prefer slightly verbose syntax over
>> shorter one when it makes reading the code easier. In IDB's case, I think
>> this is the case.
>>
>>
>>
>>>  For the latter bit, I reckon it would be a good practice for groups
>>>> working on low-level APIs to more or less systematically produce a library
>>>> that operates at a higher level. This would not only help developers in
>>>> that they could pick that up instead of the lower-level stuff, but more
>>>> importantly (at least in terms of goals) it would serve to validate that
>>>> the lower-level design is indeed appropriate for librarification.
>>>>
>>>
>>> I think that would be a good idea. Also, people making those low level
>>> APIs should still keep in mind that the resulting code should not be too
>>> verbose or complex. Librarification should be an advantage, but not a de
>>> facto requirement for developers when it comes to such APIs. It should
>>> still be feasable for them to write code in the raw low level API without
>>> writing uncomfortably verbose or complex code for simple tasks. Spec
>>> designers of low level APIs should not take this as a license to make
>>> things so complex that only they and a few others understand it, and then
>>> hope that some others will go ahead and make it simple for the 'common
>>> folk' through an abstraction library.
>>
>>
>> I quite don't see how to simplify IDB syntax much more.
>>
>
> I've avoided weighing in on this thread until I had more IDB experience.
> I've been wrestling with it on two fronts of late:
>
>
>    - A re-interpretation of the API based on Futures:
>
>    https://github.com/slightlyoff/DOMFuture/tree/master/reworked_APIs/IndexedDB
>    - A new async LocalStorage design + p(r)olyfill that's bootstrapped on
>    IDB:
>    https://github.com/slightlyoff/async-local-storage
>
> While you might be right that it's unlikely that the API can be
> "simplified", I think it's trivial to extend it in ways that make it easier
> to reason about and use.
>
> This thread started out with a discussion of what might be done to keep
> IDB's perceived mistakes from reoccurring. Here's a quick stab at both an
> outline of the mistakes and what can be done to avoid them:
>
>
>    - *Abuse of events*
>    The current IDB design models one-time operations using events. This *
>    can* make sense insofar as events can occur zero or more times in the
>    future, but it's not a natural fit. What does it mean for oncomplete to
>    happen more than once? Is that an error? Are onsuccess and onerror
>    exclusive? Can they both be dispatched for an operation? The API isn't
>    clear. Events don't lead to good design here as they don't encapsulate
>    these concerns. Similarly, event handlers don't chain. This is natural, as
>    they could be invoked multiple times (conceptually), but it's not a good
>    fit for data access. It's great that IDB as async, and events are the
>    existing DOM model for this, but IDB's IDBRequest object is calling out for
>    a different kind of abstraction. I'll submit Futures for the job, but
>    others might work (explicit callback, whatever) so long as they maintain
>    chainability + async.
>
>    - *Implicitness*
>    IDB is implicit in a number of places that cause confusion for folks
>    not intimately familiar with the contract(s) that IDB expects you to enter
>    into. First, the use of events for delivery of notifications means that
>    sequential-looking code that you might expect to have timing issues
>    doesn't. Why not? Because IDB operates in some vaguely async way; you can't
>    reason at all about events that have occurred in the past (they're not
>    values, they're points in time). I can't find anywhere in the spec that the
>    explicit gaurantees about delivery timing are noted (
>    http://www.w3.org/TR/IndexedDB/#async-api), so one could read IDB code
>    that registers two callbacks as having a temporal dead-zone: a space in
>    code where something might have happened but which your code might not have
>    a chance to hear about. I realize that in practice this isn't the case;
>    event delivery for these is asynchronous, but the soonest timing isn't
>    defined: end of turn? next turn? end-of-microtask? This means that it's
>    possible to have implementations the differ on delivery timing, astonishing
>    those who register event handlers at the wrong time. This is part DOM-ish
>    use of events for things they're not suited to and a lack of specificity in
>    the spec. Both can be fixed.
>
>    A related bit of implicitness is the transaction object. Auto-open and
>    auto-close might be virtues, but they come with costs. *When* does a
>    transaction auto-close? It's not clear from the spec; 4.2 says that a
>    transaction must be inactive when control returns to the event loop, but
>    gives no indication of what the nearest timing for that is. It's also not
>    clear how to keep a transaction "alive" across turns (a basic need), create
>    sub-transactions (a key feature of many transaction-oriented DBs), and
>    detect that a transaction object is in something other than the "active"
>    state. The last bit is particularly galling: you can have a handle to the
>    object, but users can't ask for state they might want, despite the spec
>    spending a great deal of time telling implementers that they must do this
>    and that with this bit. If there's a principle at issue, it's the idea that
>    specs -- particularly low-level APIs -- should not reserve to themselves
>    state and information that they need but for which they don't immediately
>    spot a user need. There's an obvious exception in the case of security
>    boundaries, but that's a different thing entirely. Generally speaking, if
>    you need it when writing down how your API operates, your users will too.
>    It's particularly punitive to be throwing exceptions for violations of
>    state you can't inspect but could manually cobble together from a large set
>    of events.
>
>    - *Confused collection interfaces
>    *IDB has a factory for databases and object stores and
>    allows retrieval of them by name (asynchronously, which is good)...but
>    doesn't provide a coherent Map interface onto them. By being DOM-ish and
>    not JS-ish, IDB once again creates oddball JS objects that could pun with
>    built-ins and therefore ease the learning curve, but doesn't. No, these
>    aren't (synchronous) maps, but punning the API with ES6's Map type would go
>    a long way.
>
>    - *Doubled API surface for sync version*
>    I assume I just don't understand why this choice was made, but the
>    explosion of API surface area combined with the conditional availability of
>    this version of the API make it an odd beast (to be charitable).
>
>    - *The idea that this is all going to be wrapped up by libraries anyway
>    *
>    This is aesthetic, as therefore subjective, but IDB is not a beautiful
>    API; nor does it seem evident that beauty and clarity were explicit goals.
>    I wasn't involved and don't know all the motivations (nor do I have time to
>    read all the minutes now), but there seems to be some apology happening now
>    for the lack of beauty and usability the the API based on the idea that
>    it'll just be wrapped up by libraries. This is failure for an API designer;
>    we should recognize it as such and try to belay it as long as possible.
>    Yes, all APIs are eventually wrapped as our general level of abstraction
>    goes up the stack, but it's possible to provide solid, usable APIs that
>    stand the test of time. Certainly no *new* API should plan on being as
>    painful to use as DOM has been historically.
>
> I'll close by saying that all of this is tractable. We can retrofit
> IDBRequest to be a Future subclass, create a Map-alike interface for the
> list of DB's and object stores, and move away from events where they're not
> natural; all without breaking the API. I'm hopeful we can do it quickly.
>
>
>> I think its request object based API is very nice and transactions are
>> much appreciated. Possible simplification could be achieved by introducing
>> somekind of auto transaction mechanism so that user could get and change
>> objects without creating transactions. There are some challenges to enable
>> this and it would complicate the engine especially if transactions are
>> still supported when users want to use those. And I hope transactions are
>> not dropped completely. When using CouchDB, I often find my self writing
>> some fairly painful code to handle the lack of transactions.
>>
>> Since IDB is aiming for its first standardised version of the API, I
>> wouldn't be too worried about people writing Javascript libraries that
>> simplify its use. As long as all low level capabilities are in place for
>> writing these abstractions, we should be in good order for the first
>> version of the standard. Later in following versions of the API we have
>> more experience about painful parts of IDB API and we can improve it and
>> simplify its use. Extending API by creating additional abstractions to
>> simplify its use is often more easier than going to other direction at
>> least according to my experience.
>>
>> --
>> Miko Nieminen
>> miko.nieminen@iki.fi
>> miko.nieminen@gmail.com
>>
>>