Re: IndexedDB: Syntax for specifying persistent/temporary storage

Good writeup, Jonas--I think you've hit the major points.

I think numeric priorities are both overkill and underpowered,
depending upon their specific implementation.  Without the promise
we're currently making for Persistent storage [this will never be
cleared unless you do it or the user explicitly requests it], numeric
priorities are ultimately weaker than apps want.  Unless we say that
the top priority is the same as persistent, in which case we've added
complexity without taking any away.

The idea of Default is kind of appealing, and easy to migrate over to,
but I'm not sure it's necessary.  As Kinuko says, we can just unlock
Persistent storage for apps on install, and let them migrate over
whichever data needs it.  This would work better if we supplied a tool
to do an atomic migration, though--using the current APIs, apps would
have to use 2x their storage during the transition, and browser
developers might be able to implement it internally with a simple flag
change or directory rename.

I don't have a strong opinion there, but I lean toward just the two
types rather than three.

As for Alex's "please clear up space" event--it's not clear to me how
to do that cleanly for apps that aren't currently loaded, which may
need to talk to servers that aren't currently running, which the user
may never plan to run again, or which require credentials to access
their stored data, etc.


On Wed, Dec 11, 2013 at 7:39 PM, Jonas Sicking <jonas@sicking.cc> wrote:
> Hi All,
>
> Thanks Jan for sending this.
>
> Now let me throw a giant wrench into this discussion :-)
>
> Unfortunately as we've been discussing webapps, manifests etc at
> mozilla I've slowly come to the realization that the
> temporary/persistent categorization isn't really fulfilling all the
> envisioned use cases.
>
> The background is that multiple platforms are now building the
> functionality to run normal websites outside of the browser.
>
> iOS was one of the first popular implementations of this. If you put a
> <meta name="apple-mobile-web-app-capable" content="yes"> in the markup
> of a page, and the user use "bookmark to homescreen" feature in iOS
> Safari, that almost turns the website into an "app" [1].
>
> Google is currently working on implementing the same feature in Chrome
> for Android. At mozilla we created a proposal [2] for what is
> essentially a standardized version of the same idea.
>
> I think this approach is a really awesome use of the web and something
> that I'm very interested in supporting when designing these storage
> APIs.
>
> To support this use case, I think it needs to be possible for a
> website to first start as a website which the user only has a casual
> connection with, then gradually grow into something that the user
> essentially treats as a trusted app.
>
> Such a trusted app should have much more ability to store data without
> having to ask the user for permission, or without that data being
> suddenly deleted because we're low on disk space. In short, such an
> app should be treated more like a native app when it comes to storage.
>
> There are a few ways we can enable this use case. In the discussion
> below I'll use IndexedDB as an example of storage API, but it applies
> to all storage APIs equally.
>
> A)
> The "temporary"/"persistent" split almost enables this. We could say
> that when something that's a normal website stores data in temporary
> storage we count that data towards both per-origin and global quotas.
> If the global quota fills up, then we silently delete data from
> websites in an LRU fashion.
>
> If the user converts the website to an app by using "bookmark to
> homescreen" then we simply start treating the data stored in the
> temporary storage as persistent. I.e. we don't count it towards the
> global temporary-storage quota and we never delete it in order to make
> room for other websites.
>
> For "persistent" databases we would for normal websites put up a
> prompt (I'll leave out details like if this happens only when the
> quota API is used, or if can happen when the database is being written
> to). If "persistent" storage is used by a bookmarked app we simply
> would not prompt. In neither case would data stored in persistent
> storage ever be silently deleted in order to make room for other
> storage.
>
> The problem with this solution is that it doesn't give bookmarked apps
> the ability to create truly "temporary" data. Even data that a
> bookmarked app puts in the "temporary" storage is effectively treated
> as persistent and so not deleted if we start to run low on disk space.
> Temporary storage for apps is a feature that Android has, and that to
> some extent *nix OSs has had through use of /tmp. It definite is
> something that seems nice for constrained mobile devices.
>
> B)
> We could create a "temporary"/"default"/"persistent" split. I.e. we
> create three different storage categories.
>
> The "default" is what's used if no storage category is explicitly
> specified a IDB database is created. For normal webpages "default" is
> treated like "temporary". I.e. it is counted towards the same quotas
> as temporary storage and it's deleted automatically if we run low on
> space. However once a page transitions to being an installed app data
> in "default" is treated as "persistent".
>
> "temporary" storage would always be treated as truly temporary. I.e.
> the implementation is always free to delete data there if the user is
> running low on storage. If data stored by bookmarked apps count
> towards the same quotas as temporary data stored by web pages is an
> implementation decision.
>
> "persistent" storage would behave as in A. I.e. webpages can't use it
> without there being a prompt involved at some point. Bookmarked
> webapps can use it without prompt.
>
> C)
> Rather than having different storage categories, we could simply
> introduce something like priorities. I.e. the page could set a numeric
> priority when creating a database. It could additionally use the quota
> API to request a number of MB of storage that wouldn't be deleted when
> we run low on storage.
>
> For a webpage, when we start running low on storage we delete storage
> containers, starting with the lowest priority one, until that origin's
> total storage is below the amount of storage that it had requested
> through the quota API.
>
> For a bookmarked app, we simply automatically grant it unlimited storage.
>
> This approach has the advantage that it gets rid of the split between
> different storage categories, which certainly simplifies things.
> However it also seems quite fragile.
>
> First of all it means that rather than getting rid of none or all
> "temporary" databases, the page now has to deal with different
> combinations of storage containers having been deleted.
>
> Additionally, it means that if some database that holds very important
> information grows ever so slightly larger than the amount of storage
> that it requested from the quota API, now that data can suddenly get
> deleted.
>
> Finally, it means that we're not solving the use case of wanting to
> enable bookmarked web apps to store truly temporary data that can be
> deleted if the user runs low on disk space. Though possibly we could
> do something like using negative storage priorities for temporary
> data. I.e. any priorities below 0 would be eligible for deletion even
> for bookmarked apps.
>
> D)
> Alex Russell proposed adding an event which asks a page "I'm low on
> storage, please clear out any data that is not important".
>
> This could be combined with either A or C to enable installed apps to
> have really temporary data. It has the advantage that data that isn't
> critical can be stored in the same IDB database as critical data which
> is really nice. I.e. you no longer have to separate critical and
> non-critical data into separate databases which prevent transactions
> that touch both.
>
> However it has the disadvantage that it makes clearing out temporary
> data a *really* heavy operation. It means having to launch all apps in
> the background and have them run code. This could mean spawning lots
> of processes and doing lots of complicated queries and other IO.
>
> This is probably heavy enough that it's not an option for us in Firefox OS.
>
> E)
> Your ideas here. I'd love to find better solutions than any of the above.
>
>
> So far I've only been able to think of B as solving all the use cases.
> However it feels terribly complicated to have three different storage
> categories. But it does have the advantage that for a developer that
> simply doesn't pay attention to any of this will get a pretty good
> default policy.
>
>
> [1] https://developer.apple.com/library/safari/documentation/AppleApplications/Reference/SafariHTMLRef/Articles/MetaTags.html
> [2] http://manifest.sysapps.org/
>
> / Jonas
>

Received on Friday, 13 December 2013 18:36:20 UTC