Re: [IndexedDB] Lots of small nits and clarifying questions

On Sat, Mar 13, 2010 at 9:02 AM, Nikunj Mehta <nikunj@o-micron.com> wrote:

> Thanks for your patience. Most questions below don't seem to need new spec
> text.
>
> On Feb 18, 2010, at 9:08 AM, Jeremy Orlow wrote:
>
> I'm sorry that I let so much IndexedDB feedback get backlogged.  In the
> future, I'll try to trickle things out slower.
>
> *
> *
> *Indexes:*
>
> 1) Creation of indexes really needs to be made more clear.
>
>
> Did you mean the algorithm for this process or the API documentation?
>
> For example, does creation of the index block everything until it's
> complete or does the database get created in the background?
>
>
> If the index is created using the synchronous call, then the calling worker
> will block until the index is created. If the async API is used, the rest of
> the database can continue to work and the worker should be able to continue
> to other calls without waiting for the index creation to get over.
>
>  What if I have 1gb of my mail stored in IndexedDB and then a database
> migration adds an index?
>
>
> Depends on your choice of workers and sync/async API.
>
>  Is my app completely unusable during that time?
>
>
> It doesn't have to be in a well-designed app.
>
>   What if the browser is exited half way through building (you can't just
> delete it)?
>
>
> Regardless of whether the sync or the async API, if you choose to perform
> DDL (not just index creation, but also database creation) in a transaction,
> then it could be done atomically.
>
>  What happens if you query the database while it's building in the
> background-building case (should it emulate it via entity-store-scans)?
>
>
> The index doesn't exist until it is created. If you use an index before it
> exists (even if it is in the process of being created), then the outcome
> depends on whether the index is created in a transaction. If yes, the call
> will fail because the index will not exist. If no, then the call may use
> partial index results.
>
>  These are all very important questions whose answers should be
> standardized.
>
>
> This feels like good non-normative text to me.
>

All of this seems fine for v1.


>  2) Why are Indexes in some database-global namespace rather than some
> entity-store-global namespace?
>
>
> This approach is fine by me.
>
>  I know in SQL, most people use the table name as a prefix for their index
> names to make sure they're unique.  Why inherit such silliness into
> IndexedDB?  Why not connect every index to a particular entity-store?
>
>
> Indexes in IndexDB are already dependent on an entity store.
>

Ok, then I'd suggest we move the create/open/delete index methods to the
IDBObjectStoreSync/IDBObjectStoreRequest interfaces to reflect this.


> 6) The specific ordering of elements should probably be specced including a
> mix of types.
>
>
> Can you propose spec text for this? What do you think about the text in
> http://www.w3.org/TR/IndexedDB/#key-construct?
>

If we're only adding long long for v1, then I think language similar to
what's there now is probably OK.  But now that I think about it, I'm a bit
concerned that we might be backing ourselves into a corner for the future.
 I also noticed that the sort order of JavaScript seems to order it numbers,
strings, and then nulls (not strings, numbers, nulls).

I wonder if there is some other spec on sort order we can cite rather than
rolling our own.


> *Key ranges / cursors:*
>
> 1) Is an open or closed key range the default?
>
>
> Not sure what you mean by default here?
>

The parameters are optional.  If you don't specify them, what should they
be?


>
>
> 2) What happens when data mutates while you're iterating via a cursor?
>
>
> This is covered by http://www.w3.org/TR/IndexedDB/#dfn-mode
>

That applies to two separate transactions.  As far as I can tell, it should
be possible to have a cursor open and then delete an element that the cursor
is currently traversing all within the same transaction.  Am I missing
something?


> 3) In the spec, get and getObject seem to assume that only one element can
> be returned...but that's only true if unique is true.  What do you do if
> there are multiple?
>
>
> ObjectStore doesn't allow storage of multiple values with a given key. See
> http://www.w3.org/TR/IndexedDB/#object-store-concept
>
> Multiples only make sense if you are retrieving from an Index. The
> following spec text could be added to 3.2.5 and 3.2.6 to return the first
> value in the sorted order of their keys.
>

This text should work.


>
> 3.2.5
>
> 3. Let /value/ be the value of the record first in the sorted order of
> their keys corresponding to the key /key/ in /index/
>
> 3.2.6
>
> 3. Return the value for the record first in the sorted order of their keys
> corresponding to the key /key/ in /index/
>
>
> 4) Why can the cursor only travel in one direction?
>
>
> Many things are possible. This is a scope limitation. You can open multiple
> cursors each going in different directions.
>

I was just trying to understand why.  I agree there's no super compelling
use case.

>  5) What if you modify a value that then implicitly (via the key-path)
> changes the index that your cursor is currently iterating over?
>
>
> It would be perfectly fine to remove or update a value while using a
> cursor. See
> http://dev.w3.org/2006/webapi/WebSimpleDB/#widl-IDBCursorSync-value for
> errors that may occur upon set.
>
> Specifically, if you remove the item at cursor's current position, the
> cursor stay in its position but accessing the cursor value will return null.
> You can't change the value at a cursor's current location, if the cursor is
> based on an auto-populated index. For other cursors, if you change the value
> at a cursor's current location, it will show up at its new position in the
> cursor.
>

I see that this behavior is well specced in IDBCursorSync.remove()'s
description.  But I don't think it's as clear that this is what should
happen if the value is deleted via some other mechanism, or if the value is
altered (which is what I asked a question about earlier).  I think similar
behavior (returning null until .continue() is called) is probably the best
option.


> Concurrency mode of an object store on which the cursor is used limits
> concurrent changes.
>
>
>
> *Transactions:*
>
> 1) We feel strongly that nested transactions should be allowed.  Closed
> nested transactions should be simple to implement and will make it much
> easier for multiple layers of abstraction to use IndexedDB without knowledge
> of each other.
>
>
> Can you elaborate/provide evidence for this? We have avoided nested
> transactions to keep the footprint of this API small.
>

Andy Palay has done a lot of work with Gmail offline code, gears, and
WebSQLDatabase.  These are his words in an email to me:

"""
Implicit versus explicit nested transactions. We currently use implicit
nested transactions (although the inner transactions have no effect) so that
we can write code that will maintain transaction semantics regardless of the
invocation of a given method. For example the setting of a label is
surrounded by a transaction. Within sync that is called by an method that
sets an outer transaction to handle the entire sync response. The inner
function actually has no knowledge of the outer method and thus no knowledge
of the outer/parent transaction.
"""

As I understand it, this type of pattern is fairly common inside and outside
of Google.  Because of the proliferation of cut-and-paste coding and use of
libraries that sometimes overlap functionality, it seems like a good idea to
make transactions composable.

And as far as I can tell, the API footprint wouldn't really change much spec
wise or implementation wise.  You'd just hold the transaction open until the
last .commit() is called.  (Maybe throwing an exception if the transactions
are committed out of order?)

> 2) In the spec, dynamic transactions and the difference between static and
> dynamic are not very well explained.
>
>
> Can you propose spec text?
>

In 3.1.8 of http://dev.w3.org/2006/webapi/WebSimpleDB/ in the first
paragraph, adding a sentence would probably be good enough.  "If the scope
is dynamic, the transaction may use any object stores or indexes in the
database, but if another transaction touches any of the resources in a
manner that could not be serialized by the implementation, a RECOVERABLE_ERR
exception will be thrown on commit." maybe?

>
> 3) I'm not sure that I like how the spec talks about commits being durable
> but then later says "Applications must not assume that committing the
> transaction produces an instantaneously durable result. The user agent may
> delay flushing data to durable storage until an appropriate time."
>
>
> This is also called - as durable as the underlying platform is. Nothing one
> can do here.
>
>  It seems like the language should be made more consistient.
>
>
> We have had discussion in the past about this and the group agreed at the
> time that this language is both valid and desirable.
>

I vaguely remember those.  I guess I just don't see the point of saying "A
transaction <#dfn-transaction> represents an atomic and durable set of data
access and mutation operations." when it really doesn't.


> Also, it seems like there should be some way to ensure it is durable on
> disk for when it's absolutely necessary.  (But maybe with a note that UAs
> are free to rate limit this.)
>
>
> Implementations may not be able to support this due to platform reasons and
> there seems little benefit in offering this optional feature.
>
>
>
> *Misc:*
>
> 1) Structured clone is going to change over time.  And, realistically, UAs
> won't support every type right away anyway.  What do we do when a value is
> inserted that we do not support?
>
>
> We will evolve the text as and when the same evolves in WebStorage.
>

I don't know of any implementations which have moved away from only allowing
strings within WebStorage.  I suspect that not fully supporting the
structured clone algorithm as specced is one of the reasons for this.

As far as I can tell, you're essentially saying that fully supporting the
the structured clone algorithm a pre-req for IndexedDB?  I guess I can't
argue too much with that, but I'm not sure how realistic it is.  I know we
only half support it at the moment in Chromium.

> 2) It seems that you can only be connected to one database at a time?
>
>
> Why does it so seem? A single worker (background or foreground) could be
> connected to several databases or be simultaneously connected through
> multiple connections to a single database. See
> http://dev.w3.org/2006/webapi/WebSimpleDB/#dfn-connection
>

I think I misread things here.  Sorry.


>
>  If so, why?
>
> 3) Do we have enough distinct error codes?  For example, there are multiple
> ways to get a NON_TRANSIENT_ERR when creating a transaction.  Error strings
> can help with debugging, but they can differ between UAs.  It seems as
> though all errors should be diagnosable via the error codes.
>
>
> More error codes are harder to justify. Nevertheless, NON_TRANSIENT_ERR in
> the case of transaction creation indicates bugs.
>
>
> 4) In 3.3.2, openCursor takes in an optional IDBKeyRange and then an
> optional direction.  But what if you don't want a range but you do want a
> particular direction?  Are implementations expected to handle this by
> looking at whether the first parameter is a IDBKeyRange or not?  Same goes
> for IDBIndexSync.
>
>
> I am fine with whatever mechanism passes muster with IDL lawyers.
>

Who can we ask about this?  If no one, then I think we should just specify
something and change it if/when someone complains.  I think it's a mistake
to try and do something cleaver and we should just say "undefined" indicates
"don't care" in an optional parameter.  Guidance from others would be great
though.  :-)


> 5) Similarly, put takes 2 optionals.  Depending on the object store it may
> or may not make sense for there to be a key param.  I guess the javascript
> bindings will need to have knowledge of whether a key is expected and/or
> disallow boolean keys?  It'd probably be better to avoid this from a
> bindings point of view.
>
>
> Agreed. I am open to suggestions for improving this.
>
>
> 3.2.2.4 - why would you skip the next step?
>
>
> Can you be more precise? There is no skipping in 3.2.2 step 4.
>
> 3.2.2.6 - should be preform one or the other, right?
>
>
> Can you be more precise? There is no skipping in 3.2.2.6
>
>
> 3.2.2.6.1 - should be "if it has a key generator" right?
>
>
> Can you be more precise? There is no skipping in 3.2.2.6.1
>

I'm sorry.  I apparently mixed up the numbers or the spec changed since I
wrote these down.  I guess ignore these for now.


> 3.3.2 - if createObjectStore converts a null name to the empty string, why
> woudln't openObjectStore, create/open index, and
> removeObjectStore/removeIndex?
>
>
> Perhaps it is best to remove [Null=Empty] annotations from 3.3.2
> completely. All the affected parameters are already non-optional.
>

This seems best.  From the WebIDL spec it seems these will be converted to
the string "null" which seems fine.