Re: [IndexedDB] Design Flaws: Not Stateless, Not Treating Objects As Opaque

What is the minimum that can be in IDB? I am guessing the following:

1. Sorted key-opaque value transactional store
2. Lookup of keys by values (or parts thereof)

#1 is essential.
#2 is unavoidable because you would want to efficiently manipulate values by
values as opposed to values by key.

I know of no efficient way of doing callbacks with JS. Moreover, avoiding
indices completely seems to miss the point. Yes, IDB can be used without key
paths and indices. When you do that, you would not have any headache of
setVersion since every version change either adds or removes an "object
store". Next, originally, I also had floated the idea of application managed
indices, but implementors thought of it as cruft.

On Sun, Mar 20, 2011 at 3:10 PM, Joran Greef <joran@ronomon.com> wrote:

>
> > On 20 Mar 2011, at 4:54 AM, Jonas Sicking wrote:
> >
> > I don't understand what you are saying about application state though,
> > so please do start that as a separate thread.
>
> At present, there's no way for an application to tell IDB what indexes to
> modify w.r.t. an object at the exact moment when putting or deleting that
> object. That's because this behavior is defined in advance using
> "createIndex" in a "setVersion" transaction. And then how IDB extracts the
> referenced value from the object is done using an IDB idea of "key paths".
> But right there, in defining the indexes in advance (and not when the index
> is actually modified, which is when the object itself is modified), you've
> captured application state (data relationships that should be known only to
> the application) within IDB. Because this is done in advance (because IDB
> seems to have inherited this assumption that this is just the way MySQL
> happens to do it), there's a disconnect between when the index is defined
> and when it's actually used. And because of "key paths" you now need to spec
> out all kinds of things like how to handle compound keys, multiple values.
> It's becoming a bit of a spec-fest.
>
> That this bubble of state gets captured in IDB, it also means that IDB now
> needs to provide ways of updating that captured state within IDB when it
> changes in the application (which will happen, so essentially you now have
> your indexing logic stuck in the database AND in the application and the
> application developer now has to try and keep BOTH in sync using this
> awkward pre-defined indexes interface), thus the need for a setVersion
> transaction in the first place. None of this would be necessary if the
> application could reference indexes to be modified (and created if they
> don't exist, or deleted if they would then become empty) AT THE POINT of
> putting or deleting an object. Things like data migrations would also be
> better served if this were possible since this is something the application
> would need to manage anyway. Do you follow?
>
> The application is the right place to be handling indexing logic. IDB just
> needs to provide an interface to the indexing implementation, but not handle
> extracting values from objects or deciding which indexes to modify. That's
> the domain of the application. It's a question of encapsulation. IDB is
> crossing the boundaries by demanding to know ABOUT the data stored, and not
> just providing a simple way to put an object, and a simple way to put a
> reference to an object to an index, and a simple way to query an index and
> intersect or union an index with another. Essentially an object and its
> index memberships need to be completely opaque to IDB and you are doing the
> opposite. Take a look at the BDB interface. Do you see a setVersion or
> createIndex semantic in there?


BDB has secondary databases, which are the same as indices with a one to
many relation between primary and secondary database. Moreover, BDB uses
application callbacks to let the application encapsulate the definition of
the index.


> Take a look at Redis and Tokyo and many other things. Do you see a
> setVersion or createIndex semantic in there? Do these databases have any
> idea about the contents of objects? Any concept of key paths?


I, for one, am not enamored by key paths. However, I am also morbidly aware
of the perils in JS land when using callback like mechanisms. Certainly, I
would like to hear from developers like you how you find IDB if you were to
not use any createIndex at all. Or at least that you would like to manage
your own indices.


> No, and that's the whole reason these databases were created in the first
> place. I'm sure you have read the BDB papers. Obviously this is not the
> approach of MySQL. But if IDB is trying to be MySQL but saying it wants to
> be BDB then I don't know. In any event, Firefox would be brave to also embed
> SQLite. Let the better API win.
>
> How much simpler could it be? At the end of the day, it's all objects and
> sets and sorted sets, and see Redis' epiphany on this point. IDB just needs
> to provide transactional access to these sets. The application must decide
> what goes in and out of these sets, and must be able to do it when it wants
> to, not some time in advance. I bring this up because I once wrote the exact
> same kind of database that you are writing now (where one thinks it would be
> good if the database did NOT treat objects as opaque... that the database
> should be smart about the contents of objects and share control for how
> objects relate to each other etc.) and I have since seen how much better,
> simpler, faster the alternative is. So unless you have formidable reasons
> for maintaining the status quo in light of the above, even if you don't
> understand this concept of application state getting stuck in IDB, and even
> though you advocate that WebSQL is not deprecated and that we can consider
> LocalStorage to be an alternative, then it is my hope that you will heed
> this and make something of it. I'm sorry if this is not the kind of feedback
> you want to hear at this stage, but IDB needs to be good for more than just
> HTML 5 todo list demos.
>