Re: [IndexedDB] Dynamic Transactions (WAS: Lots of small nits and clarifying questions)

On Wed, Apr 21, 2010 at 11:17 PM, Nikunj Mehta <nikunj@o-micron.com> wrote:

>
> On Apr 21, 2010, at 5:11 PM, Jeremy Orlow wrote:
>
> On Mon, Apr 19, 2010 at 11:44 PM, Nikunj Mehta <nikunj@o-micron.com>wrote:
>
>>
>> On Mar 15, 2010, at 10:45 AM, Jeremy Orlow wrote:
>>
>> On Mon, Mar 15, 2010 at 3:14 PM, Jeremy Orlow <jorlow@chromium.org>wrote:
>>
>>> On Sat, Mar 13, 2010 at 9:02 AM, Nikunj Mehta <nikunj@o-micron.com>wrote:
>>>>
>>>> On Feb 18, 2010, at 9:08 AM, Jeremy Orlow wrote:
>>>>
>>>  2) In the spec, dynamic transactions and the difference between static
>>>> and dynamic are not very well explained.
>>>>
>>>>
>>>> Can you propose spec text?
>>>>
>>>
>>> In 3.1.8 of http://dev.w3.org/2006/webapi/WebSimpleDB/ in the first
>>> paragraph, adding a sentence would probably be good enough.  "If the scope
>>> is dynamic, the transaction may use any object stores or indexes in the
>>> database, but if another transaction touches any of the resources in a
>>> manner that could not be serialized by the implementation, a RECOVERABLE_ERR
>>> exception will be thrown on commit." maybe?
>>>
>>
>> By the way, are there strong use cases for Dynamic transactions?  The more
>> that I think about them, the more out of place they seem.
>>
>>
>> Dynamic transactions are in common place use in server applications. It
>> follows naturally that client applications would want to use them.
>>
>
> There are a LOT of things that are common place in server applications that
> are not in v1 of IndexedDB.
>
>
>> Consider the use case where you want to view records in entityStore A,
>> while, at the same time, modifying another entityStore B using the records
>> in entityStore A. Unless you use dynamic transactions, you will not be able
>> to perform the two together.
>>
>
> ...unless you plan ahead.  The only thing dynamic transactions buy you is
> not needing to plan ahead about using resources.
>
>
>> The dynamic transaction case is particularly important when dealing with
>> asynchronous update processing while keeping the UI updated with data.
>>
>>
>>
>> Background: Dynamic and static are the two types of transactions in the
>> IndexedDB spec.  Static declare what resources they want access to before
>> they begin, which means that they can be implemented via objectStore level
>> locks.  Dynamic decide at commit time whether the transaction was
>> serializable.  This leaves implementations with two options:
>>
>> 1) Treat Dynamic transactions as "lock everything".
>>
>>
>> This is not consistent with the spec behavior. Locking everything is the
>> static global scope.
>>
>
> I don't understand what you're trying to say in the second sentence.  And I
> don't understand how this is inconsistent with spec behavior--it's simply
> more conservative.
>
>
>>
>>
>> 2) Implement MVCC so that dynamic transactions can operate on
>> a consistent view of data.  (At times, we'll know a transaction is doomed
>> long before commit, but we'll need to let it keep running since only
>> .commit() can raise the proper error.)
>>
>>
> MVCC is not required for dynamic transactions. MVCC is only required to
> open a database in the DETACHED_READ mode.
>

But SNAPSHOT_READ (aka detached read) is part of the spec.  So we will need
to support MVCC for v1?


> Since locks are acquired in the order in which they are requested, a
> failure could occur when an object store is being opened, but it is locked
> by another transaction. One doesn't have to wait until commit is invoked.
>

I don't see any explicit mention of how/when this would occur in the
algorithm section.  The only place in the spec I see any mention of failure
(by way of raising/returning RECOVERABLE_ERR) is in the text about
transaction's commit().

>  Am I missing something here?
>>
>>
>> If we really expect UAs to implement MVCC (or something else along those
>> lines), I would expect other more advanced transaction concepts to be
>> exposed.
>>
>>
> What precisely are you referring to? Why are these other more advanced
> transaction concepts required?
>
>   If we expect most v1 implementations to just use objectStore locks and
>> thus use option 1, then is there any reason to include Dynamic transactions?
>>
>>
> Why do you conclude that most implementations just use object store locks?
>

Isn't this assumption why the transaction model was so simplified?  Maybe
I'm remembering wrong?



On Thu, Apr 22, 2010 at 12:06 AM, Pablo Castro <Pablo.Castro@microsoft.com>
 wrote:

> On Apr 21, 2010, 11:18 PM Nikunj Mehta wrote:
>
> On Apr 21, 2010, at 5:11 PM, Jeremy Orlow wrote:
>
>
> On Mon, Apr 19, 2010 at 11:44 PM, Nikunj Mehta <nikunj@o-micron.com>
> wrote:
>
> On Mar 15, 2010, at 10:45 AM, Jeremy Orlow wrote:
>
>
> On Mon, Mar 15, 2010 at 3:14 PM, Jeremy Orlow <jorlow@chromium.org> wrote:
> On Sat, Mar 13, 2010 at 9:02 AM, Nikunj Mehta <nikunj@o-micron.com> wrote:
> On Feb 18, 2010, at 9:08 AM, Jeremy Orlow wrote:
> >> 2) In the spec, dynamic transactions and the difference between static
> and dynamic are not very well explained.
> >>
> >> Can you propose spec text?
> >>
> >> In 3.1.8 of http://dev.w3.org/2006/webapi/WebSimpleDB/ in the first
> paragraph, adding a sentence would probably be good enough.  "If the scope
> is dynamic, the transaction may use any object stores or indexes in the
> database, but if another transaction touches any of the resources in a
> manner that could not be serialized by the implementation, a RECOVERABLE_ERR
> exception will be thrown on commit." maybe?
> >>
> >> By the way, are there strong use cases for Dynamic transactions?  The
> more that I think about them, the more out of place they seem.
> >>
> >> Dynamic transactions are in common place use in server applications. It
> follows naturally that client applications would want to use them.
> >>
> >> There are a LOT of things that are common place in server applications
> that are not in v1 of IndexedDB.
> >>
> >> Consider the use case where you want to view records in entityStore A,
> while, at the same time, modifying another entityStore B using the records
> in entityStore A. Unless you use dynamic transactions, you will not be able
> to perform the two together.
> >>
> >>...unless you plan ahead.  The only thing dynamic transactions buy you is
> not needing to plan ahead about using resources.
> >>
> >> The dynamic transaction case is particularly important when dealing with
> asynchronous update processing while keeping the UI updated with data.
>
> I strongly agree that dynamic transactions are important. Funnily enough we
> were considering proposing the other extreme, and drop all the static modes
> in favor of dynamic. This is not only about being able to transport
> server-service code to the client, but more in general about supporting
> modes of operation where the complete set of objects you'll use in a
> transaction is dependent upon things you'll only find out as you process the
> transaction; this includes the particular case where your application will
> make decisions based on data on the same database, so there is no way to
> plan ahead short of locking the whole thing.
>
>
> >> 1) Treat Dynamic transactions as "lock everything".
> >>
> >> This is not consistent with the spec behavior. Locking everything is the
> static global scope.
> >>
> >> I don't understand what you're trying to say in the second sentence.
>  And I don't understand how this is inconsistent with spec behavior--it's
> simply more conservative.
>
> Of my main concerns around being overly conservative, and with the static
> locking model in general, is its impact on concurrency. While the client
> scenarios of IndexedDB don't have the same pressure for concurrency as
> server databases, things like synchronization and other background
> processing tasks do need a based level of concurrency to operate in a
> user-friendly way.
>
>
> >> 2) Implement MVCC so that dynamic transactions can operate on
> a consistent view of data.  (At times, we'll know a transaction is doomed
> long before commit, but we'll need to let it keep running since only
> .commit() can raise the proper error.)
> >>
> >> MVCC is not required for dynamic transactions. MVCC is only required to
> open a database in the DETACHED_READ mode.
> >>
> >> Since locks are acquired in the order in which they are requested, a
> failure could occur when an object store is being opened, but it is locked
> by another transaction. One doesn't have to wait until commit is invoked.
> >>
> >> Am I missing something here?
> >>
> >> If we really expect UAs to implement MVCC (or something else along those
> lines), I would expect other more advanced transaction concepts to be
> exposed.
> >>
> >> What precisely are you referring to? Why are these other more advanced
> transaction concepts required?
> >>
> >>
> >> If we expect most v1 implementations to just use objectStore locks and
> thus use option 1, then is there any reason to include Dynamic transactions?
> >>
> >> Why do you conclude that most implementations just use object store
> locks?
>
> We were actually favoring use of the dynamic pattern. Note that other than
> the failure mode (which is a separate discussion we should have), you can do
> dynamic using regular locks instead of versioning if you follow the
> two-phase protocol[1]; that still results in a serializable schedule,
> although not with point-in-time consistency.
>

But, as you said, this means your transaction could fail at pretty much any
point.  While I think many advanced web apps will be carefully programmed
and well tested, most specs work super hard to limit the number of places
where errors can occur because otherwise web apps will have nasty races that
are hard for an web developer to reproduce.


> More in general, I'm a bit worried about the number of options around
> transactions. I understand the goal of creating an "error free" model where
> once you succeed at starting a transaction you know you won't fail due to
> concurrency work, but in the process we seem to be introducing a fair number
> of new concepts, which may be equally challenging for folks. Would it be
> worth it considering resetting this to a more traditional approach were you
> only have more or less the equivalent to dynamic (in that you don't say
> anything ahead, but you're still guaranteed a serializable schedule)? The
> downside is trickier failure modes when conflicting changes happen
> concurrently, but arguably there should be a lot less of that in a client
> application.


I understand what you're saying, but it'd be really nice if we could make
sure developers don't have to choose between overly simple (and racy)
LocalStorage and a complex IndexedDB.

Maybe it's time to hash out (and add to the spec) what the actual
requirements are for IndexedDB and/or what use cases we want to support in
v1?  If we all agree that v1 is about making a high performance database
implementation at the expense of it being a good solution for most websites
(which are typically designed by cut-n-paste coding with dubious error
handling), then so be it, but we should definitely make sure everyone's on
the same page before we head in that direction.

J