Re: FileAPI use case: making an image downloading app

On Fri, Dec 17, 2010 at 3:34 PM, Gregg Tavares (wrk) <gman@google.com> wrote:
> Sorry if this has been covered before.
> I've been wanting to write an app to download images from photo sites and
> I'm wondering if this use case has been considered for the FileAPI wrt
> Directories and System.
> If I understand the current spec it seems like their are some possible
> issues.
> #1) The spec says there is a 5000 file limit or directory.

That's a conservative number I chose for performance reasons, not
because of hard limits.  While most modern filesystems officially
support far larger directories than that, I've been warned that many
bog down quite a bit by the time you hit 10000 entries.  If anyone can
point me to actual data on the major filesystems, I'm happy to bump
that number up, but I feel that we should have some limit that
generally works with decent performance on all the major platforms.
If that means you have to do some work to work around the limit, well,
you'd probably have to have done that to work around the slowness
anyway, if you wanted to write a portable app.

I see a test at
http://www.frank4dd.com/howto/various/maxfiles-per-dir.htm with much
larger numbers; you can see the curve that come out.  I'd love to see
something like that for more filesystems and fewer files.  I know NTFS
used to do N^2 work to list directories, but I presume that's been
fixed by now.

> #2) The spec requires an app to specify how much storage it needs
> I understand the desire for the limits.  What about an app being able to
> request unlimited storage and unlimited files? The UA can decide how to
> present something to the user to grant permission if they want.

Chrome has implemented an "unlimited storage" permission for its apps.
 This is really beyond the scope of this spec; I'd like to leave it
open for UAs to experiment with.  While you do have to give a hint as
to how much storage you plan to use, we really don't restrict what
actions UAs might take in response.

> Arguments against leaving it as is:
> The 5000 file limit seems arbitrary. Any app that hits that limit will
> probably require serious re-engineering to work around it. It will not only
> have to some how describe a mapping between files on a server that may not
> have that limit, it also has the issue the user might have something
> organized that way and will require the user to re-organize. I realize that
> 5000 is a large number. I'm sure the author of csh though 1700 entires in a
> glob was a reasonable limit as well. We all know how well that turned out
> :-(   It's easy to imagine a video editing app that edits
> and composites still images. If there are a few layers and 1 image per layer
> it could easily add up to more than 5000 files in a single folder.
> The size limit also has issues. For many apps the size limit will be no
> problem but for others....  Example: You make ray tracing program, it traces
> each frame and saves the files to disc. When it runs out of room, what is
> its options? (1) fail. (2) request a file system of a larger size. The
> problem is (2) will require user input. Imagine you start your render before
> you leave work expecting it to finish by the time you get in only to find
> that 2 hours after you left the UA popped up a confirmation "this app
> requests more space"

Yes, that's exactly what I'd anticipate; I personally wouldn't want an
app to be able to fill up my hard drive without getting my permission.
 However, this isn't required by the spec, and UAs are free to
experiment with UI and quota policies as they see fit.

> The same would be true for an image downloading app. You start the app to
> download a 1000 images, not knowing their sizes before hand. You start it
> and leave. Half way through the UA puts up a conformation asking for more
> space.
> Have these use cases already come up and been rejected?

Received on Monday, 10 January 2011 23:12:05 UTC