Re: IndexedDB: Syntax for specifying persistent/temporary storage

Hi,

I'm glad to hear that IDB's going to support storage types!

On Thu, Dec 12, 2013 at 12:39 PM, Jonas Sicking <jonas@sicking.cc> wrote:

> Hi All,
>
> Thanks Jan for sending this.
>
> Now let me throw a giant wrench into this discussion :-)



> Unfortunately as we've been discussing webapps, manifests etc at
> mozilla I've slowly come to the realization that the
> temporary/persistent categorization isn't really fulfilling all the
> envisioned use cases.
>
> The background is that multiple platforms are now building the
> functionality to run normal websites outside of the browser.
>
> iOS was one of the first popular implementations of this. If you put a
> <meta name="apple-mobile-web-app-capable" content="yes"> in the markup
> of a page, and the user use "bookmark to homescreen" feature in iOS
> Safari, that almost turns the website into an "app" [1].
>
> Google is currently working on implementing the same feature in Chrome
> for Android. At mozilla we created a proposal [2] for what is
> essentially a standardized version of the same idea.
>
> I think this approach is a really awesome use of the web and something
> that I'm very interested in supporting when designing these storage
> APIs.
>
> To support this use case, I think it needs to be possible for a
> website to first start as a website which the user only has a casual
> connection with, then gradually grow into something that the user
> essentially treats as a trusted app.
>
> Such a trusted app should have much more ability to store data without
> having to ask the user for permission, or without that data being
> suddenly deleted because we're low on disk space. In short, such an
> app should be treated more like a native app when it comes to storage.
>
> There are a few ways we can enable this use case. In the discussion
> below I'll use IndexedDB as an example of storage API, but it applies
> to all storage APIs equally.
>
> A)
> The "temporary"/"persistent" split almost enables this. We could say
> that when something that's a normal website stores data in temporary
> storage we count that data towards both per-origin and global quotas.
> If the global quota fills up, then we silently delete data from
> websites in an LRU fashion.
>
> If the user converts the website to an app by using "bookmark to
> homescreen" then we simply start treating the data stored in the
> temporary storage as persistent. I.e. we don't count it towards the
> global temporary-storage quota and we never delete it in order to make
> room for other websites.
>
> For "persistent" databases we would for normal websites put up a
> prompt (I'll leave out details like if this happens only when the
> quota API is used, or if can happen when the database is being written
> to). If "persistent" storage is used by a bookmarked app we simply
> would not prompt. In neither case would data stored in persistent
> storage ever be silently deleted in order to make room for other
> storage.
>

Fyi, Chrome applies similar policy to (A) for installed apps, but only when
a special permission is requested in its manifest file. For regular apps
their 'temporary' storage is still under eviction control.

The problem with this solution is that it doesn't give bookmarked apps
> the ability to create truly "temporary" data. Even data that a
> bookmarked app puts in the "temporary" storage is effectively treated
> as persistent and so not deleted if we start to run low on disk space.
> Temporary storage for apps is a feature that Android has, and that to
> some extent *nix OSs has had through use of /tmp. It definite is
> something that seems nice for constrained mobile devices.
>

As a slightly conservative variation, would it work if we just 'unlock'
persistent storage for webapps when they're 'bookmarked' or got an
installed app-like permission?  The app could check if it could use
persistent storage without requesting (by issuing quota query API, or maybe
via some event), and convert some data into persistent if they like, but UA
doesn't do so automatically.  The UA continues to evict 'temporary' data
even for 'bookmarked' webapps (UA could prioritize temporary data for
bookmarked ones if it wants).

B)
> We could create a "temporary"/"default"/"persistent" split. I.e. we
> create three different storage categories.
>
> The "default" is what's used if no storage category is explicitly
> specified a IDB database is created. For normal webpages "default" is
> treated like "temporary". I.e. it is counted towards the same quotas
> as temporary storage and it's deleted automatically if we run low on
> space. However once a page transitions to being an installed app data
> in "default" is treated as "persistent".
>
> "temporary" storage would always be treated as truly temporary. I.e.
> the implementation is always free to delete data there if the user is
> running low on storage. If data stored by bookmarked apps count
> towards the same quotas as temporary data stored by web pages is an
> implementation decision.
>
> "persistent" storage would behave as in A. I.e. webpages can't use it
> without there being a prompt involved at some point. Bookmarked
> webapps can use it without prompt.
>

This sounds kinda reasonable to me, but I'd imagine some webapps (who are
serious about their storage) would probably want to know if the "default"
is either temporary or persistent in this case, and we may need a tiny API
for that.


> C)
> Rather than having different storage categories, we could simply
> introduce something like priorities. I.e. the page could set a numeric
> priority when creating a database. It could additionally use the quota
> API to request a number of MB of storage that wouldn't be deleted when
> we run low on storage.
>
> For a webpage, when we start running low on storage we delete storage
> containers, starting with the lowest priority one, until that origin's
> total storage is below the amount of storage that it had requested
> through the quota API.
>
> For a bookmarked app, we simply automatically grant it unlimited storage.
>
> This approach has the advantage that it gets rid of the split between
> different storage categories, which certainly simplifies things.
> However it also seems quite fragile.
>
> First of all it means that rather than getting rid of none or all
> "temporary" databases, the page now has to deal with different
> combinations of storage containers having been deleted.
>
> Additionally, it means that if some database that holds very important
> information grows ever so slightly larger than the amount of storage
> that it requested from the quota API, now that data can suddenly get
> deleted.
>
> Finally, it means that we're not solving the use case of wanting to
> enable bookmarked web apps to store truly temporary data that can be
> deleted if the user runs low on disk space. Though possibly we could
> do something like using negative storage priorities for temporary
> data. I.e. any priorities below 0 would be eligible for deletion even
> for bookmarked apps.
>

My general feeling is: priorities could be powerful if they're used in a
consistent way, but otherwise it'd just make things too complicated both
for UA implementors and webapp developers, and often that's the case in
real world.  It also might not really provide the desirable benefits (i.e.,
could be fragile).  Assigning higher priority wouldn't guarantee anything,
and how they're assigned would largely vary by apps.

D)
> Alex Russell proposed adding an event which asks a page "I'm low on
> storage, please clear out any data that is not important".
>
> This could be combined with either A or C to enable installed apps to
> have really temporary data. It has the advantage that data that isn't
> critical can be stored in the same IDB database as critical data which
> is really nice. I.e. you no longer have to separate critical and
> non-critical data into separate databases which prevent transactions
> that touch both.



> However it has the disadvantage that it makes clearing out temporary
> data a *really* heavy operation. It means having to launch all apps in
> the background and have them run code. This could mean spawning lots
> of processes and doing lots of complicated queries and other IO.
>

I'm actually a fan of this idea, as it gives most flexible option (but with
some cost).
I wouldn't say it doesn't make the eviction operation heavy, but UA could
probably do smarter than launching all apps at once.  Most apps may not be
interested in such an event at all, and UA could only check the apps that
have shown interest on adjusting their data on their own, probably via some
registration mechanism.  UA could also use some order like LRU to launch
apps one by one until it can get enough available space.

This is probably heavy enough that it's not an option for us in Firefox OS.
>
> E)
> Your ideas here. I'd love to find better solutions than any of the above.
>
> So far I've only been able to think of B as solving all the use cases.
> However it feels terribly complicated to have three different storage
> categories. But it does have the advantage that for a developer that
> simply doesn't pay attention to any of this will get a pretty good
> default policy.
>
>
> [1]
> https://developer.apple.com/library/safari/documentation/AppleApplications/Reference/SafariHTMLRef/Articles/MetaTags.html
> [2] http://manifest.sysapps.org/
>
> / Jonas
>
>