Re: IndexedDB: Syntax for specifying persistent/temporary storage

Hi All,

Thanks Jan for sending this.

Now let me throw a giant wrench into this discussion :-)

Unfortunately as we've been discussing webapps, manifests etc at
mozilla I've slowly come to the realization that the
temporary/persistent categorization isn't really fulfilling all the
envisioned use cases.

The background is that multiple platforms are now building the
functionality to run normal websites outside of the browser.

iOS was one of the first popular implementations of this. If you put a
<meta name="apple-mobile-web-app-capable" content="yes"> in the markup
of a page, and the user use "bookmark to homescreen" feature in iOS
Safari, that almost turns the website into an "app" [1].

Google is currently working on implementing the same feature in Chrome
for Android. At mozilla we created a proposal [2] for what is
essentially a standardized version of the same idea.

I think this approach is a really awesome use of the web and something
that I'm very interested in supporting when designing these storage
APIs.

To support this use case, I think it needs to be possible for a
website to first start as a website which the user only has a casual
connection with, then gradually grow into something that the user
essentially treats as a trusted app.

Such a trusted app should have much more ability to store data without
having to ask the user for permission, or without that data being
suddenly deleted because we're low on disk space. In short, such an
app should be treated more like a native app when it comes to storage.

There are a few ways we can enable this use case. In the discussion
below I'll use IndexedDB as an example of storage API, but it applies
to all storage APIs equally.

A)
The "temporary"/"persistent" split almost enables this. We could say
that when something that's a normal website stores data in temporary
storage we count that data towards both per-origin and global quotas.
If the global quota fills up, then we silently delete data from
websites in an LRU fashion.

If the user converts the website to an app by using "bookmark to
homescreen" then we simply start treating the data stored in the
temporary storage as persistent. I.e. we don't count it towards the
global temporary-storage quota and we never delete it in order to make
room for other websites.

For "persistent" databases we would for normal websites put up a
prompt (I'll leave out details like if this happens only when the
quota API is used, or if can happen when the database is being written
to). If "persistent" storage is used by a bookmarked app we simply
would not prompt. In neither case would data stored in persistent
storage ever be silently deleted in order to make room for other
storage.

The problem with this solution is that it doesn't give bookmarked apps
the ability to create truly "temporary" data. Even data that a
bookmarked app puts in the "temporary" storage is effectively treated
as persistent and so not deleted if we start to run low on disk space.
Temporary storage for apps is a feature that Android has, and that to
some extent *nix OSs has had through use of /tmp. It definite is
something that seems nice for constrained mobile devices.

B)
We could create a "temporary"/"default"/"persistent" split. I.e. we
create three different storage categories.

The "default" is what's used if no storage category is explicitly
specified a IDB database is created. For normal webpages "default" is
treated like "temporary". I.e. it is counted towards the same quotas
as temporary storage and it's deleted automatically if we run low on
space. However once a page transitions to being an installed app data
in "default" is treated as "persistent".

"temporary" storage would always be treated as truly temporary. I.e.
the implementation is always free to delete data there if the user is
running low on storage. If data stored by bookmarked apps count
towards the same quotas as temporary data stored by web pages is an
implementation decision.

"persistent" storage would behave as in A. I.e. webpages can't use it
without there being a prompt involved at some point. Bookmarked
webapps can use it without prompt.

C)
Rather than having different storage categories, we could simply
introduce something like priorities. I.e. the page could set a numeric
priority when creating a database. It could additionally use the quota
API to request a number of MB of storage that wouldn't be deleted when
we run low on storage.

For a webpage, when we start running low on storage we delete storage
containers, starting with the lowest priority one, until that origin's
total storage is below the amount of storage that it had requested
through the quota API.

For a bookmarked app, we simply automatically grant it unlimited storage.

This approach has the advantage that it gets rid of the split between
different storage categories, which certainly simplifies things.
However it also seems quite fragile.

First of all it means that rather than getting rid of none or all
"temporary" databases, the page now has to deal with different
combinations of storage containers having been deleted.

Additionally, it means that if some database that holds very important
information grows ever so slightly larger than the amount of storage
that it requested from the quota API, now that data can suddenly get
deleted.

Finally, it means that we're not solving the use case of wanting to
enable bookmarked web apps to store truly temporary data that can be
deleted if the user runs low on disk space. Though possibly we could
do something like using negative storage priorities for temporary
data. I.e. any priorities below 0 would be eligible for deletion even
for bookmarked apps.

D)
Alex Russell proposed adding an event which asks a page "I'm low on
storage, please clear out any data that is not important".

This could be combined with either A or C to enable installed apps to
have really temporary data. It has the advantage that data that isn't
critical can be stored in the same IDB database as critical data which
is really nice. I.e. you no longer have to separate critical and
non-critical data into separate databases which prevent transactions
that touch both.

However it has the disadvantage that it makes clearing out temporary
data a *really* heavy operation. It means having to launch all apps in
the background and have them run code. This could mean spawning lots
of processes and doing lots of complicated queries and other IO.

This is probably heavy enough that it's not an option for us in Firefox OS.

E)
Your ideas here. I'd love to find better solutions than any of the above.


So far I've only been able to think of B as solving all the use cases.
However it feels terribly complicated to have three different storage
categories. But it does have the advantage that for a developer that
simply doesn't pay attention to any of this will get a pretty good
default policy.


[1] https://developer.apple.com/library/safari/documentation/AppleApplications/Reference/SafariHTMLRef/Articles/MetaTags.html
[2] http://manifest.sysapps.org/

/ Jonas

Received on Thursday, 12 December 2013 03:40:02 UTC