Re: FileAPI use case: making an image downloading app

On 12/17/2010 5:03 PM, Gregg Tavares (wrk) wrote:
>
>
> On Fri, Dec 17, 2010 at 4:16 PM, Charles Pritchard <chuck@jumis.com 
> <mailto:chuck@jumis.com>> wrote:
>
>     We're actively developing such functionality.
>
>     The limit per directory is for the sake of the os file system. If
>     you want to create a data store, use indexedDB or use standard
>     file system practices (create a subdirectory tree).
>
>
> I think you're missing the point. If I have a folder with 6000 files 
> on some server and I want to mirror that through the FileAPI to the 
> user's local machine I'm screwed. I can't **mirror** it if it's not a 
> mirror.
I'm not missing the point. I'm actively developing an app that downloads 
images from photo sites.

A strict Mirror (lets use a capital M) is something you're not going to 
pull off with the File System API at this time.
You can't set meta data, like permissions/flags and 
modification/creation dates. Developing a Mirror is not feasible with 
the current API.
You can't create a direct Mirror, one which would work seamlessly with 
"rsync".

You can "mirror", the data on a remote server, and check that the data 
already exists on your file system,
by using a subdirectory system, much like the two level directory 
structure that many cache apps use (like Squid).

>     Disk space availability (quota) is an issue no matter what
>     happens. When downloading 1000 images, you'll still only be doing
>     so x at a time.
>
>
> I don't see the point you're trying to make here. I don't know the 
> size of the images before hand. Many internet APIs make getting the 
> sizes prohibitively expensive. One REST call per file. So before I can 
> download a single file I'd have to issue 1000 REST XHRs to find the 
> sizes of the files (waiting the several minutes for that to complete) 
> before I can ask the user for the space needed.  That's not a good 
> user experience. If on the other hand the user can give me permission 
> for unlimited space then I can just start downloading the files 
> without having to find out their ultimate size.
>
> I suppose I can just request 1 terabyte up front. Or query how much 
> space is free and ask for all of it.
Yes, that's correct, you'd hope the space is available.
When you run out of space, you can use a limited amount of RAM while 
waiting.

There are few resource management APIs available for memory/bandwidth 
hungry applications.

My point was that these questions you're bringing up are common to all 
cross-platform applications.

Regarding your issue of "a ray tracing program" : You'd also want to 
either: create sub directories,
or simply create a large file with its own methods.

>
>     This issues are inherent in the design of any application of
>     scale. At this point, the file system API does work for the use
>     case you're describing.
>
>     It'd be nice to see Blob storage better integrated with Web
>     Storage Apis. Ian has already spoken to this, but no followers yet
>     (afaik).
>
>     -Charles
>
>
>
>     On Dec 17, 2010, at 3:34 PM, "Gregg Tavares (wrk)"
>     <gman@google.com <mailto:gman@google.com>> wrote:
>
>     > Sorry if this has been covered before.
>     >
>     > I've been wanting to write an app to download images from photo
>     sites and I'm wondering if this use case has been considered for
>     the FileAPI wrt Directories and System.
>     >
>     > If I understand the current spec it seems like their are some
>     possible issues.
>     >
>     > #1) The spec says there is a 5000 file limit or directory.
>     >
>     > #2) The spec requires an app to specify how much storage it needs
>     >
>     > I understand the desire for the limits.  What about an app being
>     able to request unlimited storage and unlimited files? The UA can
>     decide how to present something to the user to grant permission if
>     they want.
>     >
>     > Arguments against leaving it as is:
>     >
>     > The 5000 file limit seems arbitrary. Any app that hits that
>     limit will probably require serious re-engineering to work around
>     it. It will not only have to some how describe a mapping between
>     files on a server that may not have that limit, it also has the
>     issue the user might have something organized that way and will
>     require the user to re-organize. I realize that 5000 is a large
>     number. I'm sure the author of csh though 1700 entires in a glob
>     was a reasonable limit as well. We all know how well that turned
>     out :-(   It's easy to imagine a video editing app that edits and
>     composites still images. If there are a few layers and 1 image per
>     layer it could easily add up to more than 5000 files in a single
>     folder.
>     >
>     > The size limit also has issues. For many apps the size limit
>     will be no problem but for others....  Example: You make ray
>     tracing program, it traces each frame and saves the files to disc.
>     When it runs out of room, what is its options? (1) fail. (2)
>     request a file system of a larger size. The problem is (2) will
>     require user input. Imagine you start your render before you leave
>     work expecting it to finish by the time you get in only to find
>     that 2 hours after you left the UA popped up a confirmation "this
>     app requests more space"
>     >
>     > The same would be true for an image downloading app. You start
>     the app to download a 1000 images, not knowing their sizes before
>     hand. You start it and leave. Half way through the UA puts up a
>     conformation asking for more space.
>     >
>     > Have these use cases already come up and been rejected?
>     >
>     >
>     >
>
>