Re: IndexedDB: Syntax for specifying persistent/temporary storage

Hi Kinuko and Eric,

Thanks for providing feedback from a Google perspective! Definitely
still feels like there's a lot of possibilities and options here. No
combination of solutions seem obviously best to me, so please do take
the below as me thinking out loud rather than having strong opinions.

On Wed, Dec 11, 2013 at 10:53 PM, Kinuko Yasuda <kinuko@chromium.org> wrote:
> Fyi, Chrome applies similar policy to (A) for installed apps, but only when
> a special permission is requested in its manifest file. For regular apps
> their 'temporary' storage is still under eviction control.

Interesting!

I had hoped that we would be able to align on storage policies for
installed/bookmarked hosted apps (not packaged apps). Maybe that was
too naive :-)

Though sounds like we still are very close. Close enough that it we
can probably align on API.

>> The problem with this solution is that it doesn't give bookmarked apps
>> the ability to create truly "temporary" data. Even data that a
>> bookmarked app puts in the "temporary" storage is effectively treated
>> as persistent and so not deleted if we start to run low on disk space.
>> Temporary storage for apps is a feature that Android has, and that to
>> some extent *nix OSs has had through use of /tmp. It definite is
>> something that seems nice for constrained mobile devices.
>
> As a slightly conservative variation, would it work if we just 'unlock'
> persistent storage for webapps when they're 'bookmarked' or got an installed
> app-like permission?  The app could check if it could use persistent storage
> without requesting (by issuing quota query API, or maybe via some event),
> and convert some data into persistent if they like, but UA doesn't do so
> automatically.  The UA continues to evict 'temporary' data even for
> 'bookmarked' webapps (UA could prioritize temporary data for bookmarked ones
> if it wants).

I think this would work, but it has downsides. See my answer below to Eric.

>> B)
>> We could create a "temporary"/"default"/"persistent" split. I.e. we
>> create three different storage categories.
>>
>> The "default" is what's used if no storage category is explicitly
>> specified a IDB database is created. For normal webpages "default" is
>> treated like "temporary". I.e. it is counted towards the same quotas
>> as temporary storage and it's deleted automatically if we run low on
>> space. However once a page transitions to being an installed app data
>> in "default" is treated as "persistent".
>>
>> "temporary" storage would always be treated as truly temporary. I.e.
>> the implementation is always free to delete data there if the user is
>> running low on storage. If data stored by bookmarked apps count
>> towards the same quotas as temporary data stored by web pages is an
>> implementation decision.
>>
>> "persistent" storage would behave as in A. I.e. webpages can't use it
>> without there being a prompt involved at some point. Bookmarked
>> webapps can use it without prompt.
>
> This sounds kinda reasonable to me, but I'd imagine some webapps (who are
> serious about their storage) would probably want to know if the "default" is
> either temporary or persistent in this case, and we may need a tiny API for
> that.

Definitely, if we go with this solution we should provide a way to
know what policy is used for "default".

>> D)
>> Alex Russell proposed adding an event which asks a page "I'm low on
>> storage, please clear out any data that is not important".
>>
>> This could be combined with either A or C to enable installed apps to
>> have really temporary data. It has the advantage that data that isn't
>> critical can be stored in the same IDB database as critical data which
>> is really nice. I.e. you no longer have to separate critical and
>> non-critical data into separate databases which prevent transactions
>> that touch both.
>>
>> However it has the disadvantage that it makes clearing out temporary
>> data a *really* heavy operation. It means having to launch all apps in
>> the background and have them run code. This could mean spawning lots
>> of processes and doing lots of complicated queries and other IO.
>
> I'm actually a fan of this idea, as it gives most flexible option (but with
> some cost).
> I wouldn't say it doesn't make the eviction operation heavy, but UA could
> probably do smarter than launching all apps at once.  Most apps may not be
> interested in such an event at all, and UA could only check the apps that
> have shown interest on adjusting their data on their own, probably via some
> registration mechanism.  UA could also use some order like LRU to launch
> apps one by one until it can get enough available space.

One of the situations when we need to clear out temporary storage is
when we're in the middle of a IDB (or presumably WebSQL) transaction
and run low on disk space. The page might already have been granted
permanent storage so we don't want to simply abort the transaction and
blame quota.

Would a notification such as the one described here help in that
situation? I.e. would you really want to pause the current transaction
and start firing up websites?

Or would you be forced to signal a IO error and then start firing
notifications to clear out disk space? And hope that the webpage will
attempt to write again in a few minutes?

Or would we simply hope that we can prevent getting into this
situation by firing notifications before we actually run out of disk
space?

Like you, I do really like the flexibility of the notification though.
It feels like websites are going to be more willing to tread data as
temporary if they aren't forced to keep the data in a separate
database. However I suspect that it will be hard to get this system to
really work well in the emergency
low-on-diskspace-and-someone-tries-to-write-data situation.

It's always tempting to fall back to the "let's add all options"
solution and add both "default" storage as well as a notification.


On Fri, Dec 13, 2013 at 10:35 AM, Eric U <ericu@google.com> wrote:
> Good writeup, Jonas--I think you've hit the major points.
>
> I think numeric priorities are both overkill and underpowered,
> depending upon their specific implementation.  Without the promise
> we're currently making for Persistent storage [this will never be
> cleared unless you do it or the user explicitly requests it], numeric
> priorities are ultimately weaker than apps want.  Unless we say that
> the top priority is the same as persistent, in which case we've added
> complexity without taking any away.

Agreed. Lets axe this option as no-one seems to like it.

> The idea of Default is kind of appealing, and easy to migrate over to,
> but I'm not sure it's necessary.  As Kinuko says, we can just unlock
> Persistent storage for apps on install, and let them migrate over
> whichever data needs it.  This would work better if we supplied a tool
> to do an atomic migration, though--using the current APIs, apps would
> have to use 2x their storage during the transition, and browser
> developers might be able to implement it internally with a simple flag
> change or directory rename.

If most apps are doing this migration, then I think we've created a
system that is more complicated for authors. That said, I'm really not
sure that doing this migration is something that most apps want to do.
It might very well be the case that most apps are happy to leave data
in the temporary storage.

The strongest indicator I think that once an app is "installed" that
the most common user expectation is likely that they will not lose the
data associated with that app. I.e. if a user installs a game and
plays through a few levels, then the user likely expects those levels
to be beaten when he/she picks up the game again.

Of course, in a perfect world that progress would be automatically
synchronized to a server so that those levels are beaten even if the
user loses the device and buys a new one. But we don't really have the
identity solutions for the web to support that in a smooth way yet :(

Another problem that the migration would cause is related to the
current filesystem API proposals. Both the google and mozilla
proposals use separate URLs for temporary and persistent data. I.e.
something like filesystem:http://domain.com/persistent/path/file.txt.
If apps migrate data back and forth between temporary and persistent
storage then the app has to keep track of where the data lives and
adjust which URLs it uses to accommodate that.

This is particularly tricky since some of the use cases for the
filesystem URLs that we've heard is being able to point to the
"correct" URL synchronously from within a script. Which prevents the
app from using an async storage mechanism like WebSQL or IndexedDB to
track where the data is stored.

Another way to look at this is, is it more complexity to introduce an
atomic-move API than a "default" storage area?

> As for Alex's "please clear up space" event--it's not clear to me how
> to do that cleanly for apps that aren't currently loaded, which may
> need to talk to servers that aren't currently running, which the user
> may never plan to run again, or which require credentials to access
> their stored data, etc.

The way to do this would be through service workers. The API doesn't
exist yet, but if would fit the service worker model fairly well, so I
don't think the API part here would present any big problems.

/ Jonas

Received on Tuesday, 17 December 2013 08:09:08 UTC