Re: File API - where are the missing parts?

Thanks for starting this thread, Florian. I'm in broad agreement.

On Tue, Feb 23, 2016 at 7:12 AM, Florian Bösch <pyalot@gmail.com> wrote:

> On Tue, Feb 23, 2016 at 2:48 AM, Jonas Sicking <jonas@sicking.cc> wrote:
>
>> Is the last bullet here really accurate? How can you use existing APIs to
>> listen to file modifications?
>>
> I have not tested this on all UAs, but in Google Chrome what you can do is
> to set an interval to check a files.lastModified date, and if a
> modification is detected, read it in again with a FileReader and that works
> fine.
>

Huh... we should probably specify and/or fix that.


>
>
>> There are also APIs implemented in several browsers for opening a whole
>> directory of files from a webpage. This has been possible for some time in
>> Chrome, and support was also recently added to Firefox and Edge. I'm not
>> sure how interoperable these APIs are across browsers though :(
>>
>
IIRC, Edge's API[1] mimics Chrome's, and you can polyfill Firefox's API [2]
on top of Chrome/Edge's[3]. So in theory if Firefox's pleases developers
they can adopt the polyfill, and other browsers can transition to native
support.

[1] https://lists.w3.org/Archives/Public/public-wicg/2015Sep/0000.html
[2] https://wicg.github.io/directory-upload/proposal.html
[3] https://github.com/WICG/directory-upload/blob/gh-pages/polyfill.js

... or just read Ali's excellent summary:

https://lists.w3.org/Archives/Public/public-webapps/2015AprJun/0245.html

(But that's all a tangent to Florian's main use cases...)



> There does not seem to be a standard about this, or is there? It's an
> essential functionality to be able to import OBJ and Collada files because
> they are composites of the main file and other files (such as material
> definitions or textures).
>
>
>> Another important missing capability is the ability to modify an existing
>> file. I.e. write 10 bytes in the middle of a 1GB file, without having to
>> re-write the whole 1GB to disk.
>>
> Good point
>
>
>> However, before getting into such details, it is very important when
>> discussing read/writing is to be precise about which files can be
>> read/written.
>>
>> For example IndexedDB supports storing File (and Blob) objects inside
>> IndexedDB. You can even get something very similar to incremental
>> reads/writes by reading/writing slices.
>>
>> Here's a couple of libraries which implement filesystem APIs, which
>> support incremental reading and writing, on top of IndexedDB:
>>
>> https://github.com/filerjs/filer
>> https://github.com/ebidel/idb.filesystem.js
>>
>> However, IndexedDB, and thus any libraries built on top of it, only
>> supports reading and writing files inside a origin-specific
>> browser-sandboxed directory.
>>
>> This is also true for the the Filesystem API implemented in Google Chrome
>> APIs that you are linking to. And it applies to the Filesystem API proposal
>> at [1].
>>
>> Writing files outside of any sandboxes requires not just an API
>> specification, but also some sane, user understandable UX.
>>
>> So, to answer your questions, I would say:
>>
>> The APIs that you are linking to does not in fact meet the use cases that
>> you are pointing to in your examples. Neither does [1], which is the
>> closest thing that we have to a successor.
>>
>> The reason that no work has been done to meet the use cases that you are
>> referring to, is that so far no credible solutions have been proposed for
>> the UX problem. I.e. how do we make it clear to the user that they are
>> granting access to the webpage to overwrite the contents of a file.
>>
>> [1] http://w3c.github.io/filesystem-api/
>>
> To be clear, I'm referring specifically to the ability of a user to pick
> any destination on his mass-storage device to manage his data. This might
> not be as sexy and easy as IndexDB & Co. but it's an essential
> functionality for users to be able to organize their files to where they
> want to have them, with the minimum of fuss.
>
>
Yep, use cases acknowledged. I summarize them as:

* "build a file editor" - you can't even build a non-terrible Notepad
today, since "File > (Re)Save" and "File > Save As..." aren't supported,
let alone performant (random access writes)

* "build an IDE" - the above plus directory enumeration, file/directory
watching, non-intrusive open/save.

I agree these use cases are important, and I would like the platform to
eventually support them both for native and sandboxed filesystems.

I'm aware that there's thorny questions regarding UX (although UX itself is
> rarely if ever specified in a W3C standard is it?).
>

True, but if we determine that permissions must be granted then the API
needs to be designed to handle it, e.g. entry points to the API surface are
through a requestPermission() API, everything is async, etc.


> But that does not impact all missing pieces. Notably not these:
>
>    - Save a file incrementally (and with the ability to abort): not a UX
>    problem because the mechanism to save files exists, it's just
>    insufficiently specified to allow for streaming writes.
>
>  Agreed. Streams + Blobs needs to be sorted.

>
>    - Save as pickable destination: also not a UX problem, the standard
>    solution here is to present the user with a standard operating system
>    specific file save dialog.
>
> Agreed that "File.saveAs()" or something should be non-controversial. It
seems like a new capability with minimal possibility for user confusion.
That said, IIRC most browsers supported this in the past for downloads and
have gradually shifted to a model where the user is not prompted. There may
be history here we should dig into.

One concern is: what capabilities are granted by this action? Can the
web-app re-save the file? Can it re-read the file? Does that permission
persist across sessions? For example, if I save a document template from a
site I would not expect the site to be able to read the file after I've
edited with an unrelated file editor.

>
>    - Save many files to a user pickable folder: same as above
>    - Working directory: this is more something that would go on in the
>    background of a UA, it would have to establish a "working directory" per
>    tab rather than UA-wide. No UX issues with that.
>
> Agreed. Likely doesn't even need to be specified - it'd just be a "least
surprise" behavior by the UA.


> Additionally this should be minimally UX controversial:
>
>    - Overwrite a file (either previously saved or opened): I think it'd
>    be a legitimate implementation of the UX to show an appropriate dialog at
>    the time of overwrite that indicates what is overwritten, it's just a
>    fast-track to save as pick file (and the UX can be improved by persistence
>    of choice if that is deemed an acceptable risk).
>
> So it doesn't strike me that these missing features would create massive
> UX problems, indeed, most of them create no UX problem at all.
>

There are other twists to permissions. To enumerate a few:

* Since "File > Open" is supported today (via <input type=file>) we must be
careful about exposing functionality that has similar UX (i.e. a native
file open dialog) but that implicitly grants extra permissions (e.g. being
able to modify the file). This points to either additional UX during the
action, UX when the app wants to write, or a more general permission
granted to the origin for some scope (file? directory?).

* Should permissions persist? If you're working in an editor and reload the
tab, being hit with a flurry of permission prompts is less than ideal. But
if you visit it again in a day or a year? And, similar to the "template"
case above, what if you use a web-based editor to modify a file, then
revisit the site a year later.

Again, I believe this is tractable, we just need to work through these with
UX+Privacy+Security glasses on.

I've been having private brainstorms with other folks in this area. (And
those folks should chime in here if they want to share their thoughts!)
Nothing concrete to share yet, but I'm glad to hear that others are
interested in this space. So again, thanks for kicking this off Florian!

I'm also very interested in hearing from other browser implementers; Chrome
is in the odd position of having made investments in related areas
(FileSystem API and FileWriter API) that did not see adoption in other
browsers, which is a disincentive to proceed here.