Re: FileAPI use case: making an image downloading app

On 18 December 2010 17:38, Charles Pritchard <chuck@jumis.com> wrote:

>  On 12/17/2010 5:03 PM, Gregg Tavares (wrk) wrote:
>
>
>
> On Fri, Dec 17, 2010 at 4:16 PM, Charles Pritchard <chuck@jumis.com>wrote:
>
>> We're actively developing such functionality.
>>
>> The limit per directory is for the sake of the os file system. If you want
>> to create a data store, use indexedDB or use standard file system practices
>> (create a subdirectory tree).
>>
>
>  I think you're missing the point. If I have a folder with 6000 files on
> some server and I want to mirror that through the FileAPI to the user's
> local machine I'm screwed. I can't **mirror** it if it's not a mirror.
>
> I'm not missing the point. I'm actively developing an app that downloads
> images from photo sites.
>
> A strict Mirror (lets use a capital M) is something you're not going to
> pull off with the File System API at this time.
> You can't set meta data, like permissions/flags and modification/creation
> dates. Developing a Mirror is not feasible with the current API.
> You can't create a direct Mirror, one which would work seamlessly with
> "rsync".
>
> You can "mirror", the data on a remote server, and check that the data
> already exists on your file system,
> by using a subdirectory system, much like the two level directory structure
> that many cache apps use (like Squid).
>
>   Disk space availability (quota) is an issue no matter what happens. When
>> downloading 1000 images, you'll still only be doing so x at a time.
>>
>
>  I don't see the point you're trying to make here. I don't know the size
> of the images before hand. Many internet APIs make getting the sizes
> prohibitively expensive. One REST call per file. So before I can download a
> single file I'd have to issue 1000 REST XHRs to find the sizes of the files
> (waiting the several minutes for that to complete) before I can ask the user
> for the space needed.  That's not a good user experience. If on the other
> hand the user can give me permission for unlimited space then I can just
> start downloading the files without having to find out their ultimate size.
>
>  I suppose I can just request 1 terabyte up front. Or query how much space
> is free and ask for all of it.
>
> Yes, that's correct, you'd hope the space is available.
> When you run out of space, you can use a limited amount of RAM while
> waiting.
>
> There are few resource management APIs available for memory/bandwidth
> hungry applications.
>
> My point was that these questions you're bringing up are common to all
> cross-platform applications.
>
> Regarding your issue of "a ray tracing program" : You'd also want to
> either: create sub directories,
> or simply create a large file with its own methods.
>
>
>> This issues are inherent in the design of any application of scale. At
>> this point, the file system API does work for the use case you're
>> describing.
>>
>> It'd be nice to see Blob storage better integrated with Web Storage Apis.
>> Ian has already spoken to this, but no followers yet (afaik).
>>
>> -Charles
>>
>>
>>
>> On Dec 17, 2010, at 3:34 PM, "Gregg Tavares (wrk)" <gman@google.com>
>> wrote:
>>
>> > Sorry if this has been covered before.
>> >
>> > I've been wanting to write an app to download images from photo sites
>> and I'm wondering if this use case has been considered for the FileAPI wrt
>> Directories and System.
>> >
>> > If I understand the current spec it seems like their are some possible
>> issues.
>> >
>> > #1) The spec says there is a 5000 file limit or directory.
>> >
>> > #2) The spec requires an app to specify how much storage it needs
>> >
>> > I understand the desire for the limits.  What about an app being able to
>> request unlimited storage and unlimited files? The UA can decide how to
>> present something to the user to grant permission if they want.
>> >
>> > Arguments against leaving it as is:
>> >
>> > The 5000 file limit seems arbitrary. Any app that hits that limit will
>> probably require serious re-engineering to work around it. It will not only
>> have to some how describe a mapping between files on a server that may not
>> have that limit, it also has the issue the user might have something
>> organized that way and will require the user to re-organize. I realize that
>> 5000 is a large number. I'm sure the author of csh though 1700 entires in a
>> glob was a reasonable limit as well. We all know how well that turned out
>> :-(   It's easy to imagine a video editing app that edits and composites
>> still images. If there are a few layers and 1 image per layer it could
>> easily add up to more than 5000 files in a single folder.
>> >
>> > The size limit also has issues. For many apps the size limit will be no
>> problem but for others....  Example: You make ray tracing program, it traces
>> each frame and saves the files to disc. When it runs out of room, what is
>> its options? (1) fail. (2) request a file system of a larger size. The
>> problem is (2) will require user input. Imagine you start your render before
>> you leave work expecting it to finish by the time you get in only to find
>> that 2 hours after you left the UA popped up a confirmation "this app
>> requests more space"
>> >
>> > The same would be true for an image downloading app. You start the app
>> to download a 1000 images, not knowing their sizes before hand. You start it
>> and leave. Half way through the UA puts up a conformation asking for more
>> space.
>> >
>> > Have these use cases already come up and been rejected?
>> >
>> >
>> >
>>
>
>
>
I agree with the main point, a directory tree is the only cross platform way
to do this using the filesystem, and I would have thought IndexedDB ideal
for this kind of thing too.

>The limit per directory is for the sake of the os file system.

Such limitations are much higher than this, for Linux it _used_ to be 32,000
but now is practically unlimited. For FAT32 its 65k, for NTFS 4g. 6000 seems
a bit on the low side... Which OS has such a low limit?

To me it would seem safe to increase this limit to 32,000.


Cheers,
Keean.