Re: [w3c/FileAPI] Define chunk size for `ReadableStream` created by `blob::stream()` (#144)

So here is an in-depth explanation.

I'm making a wasm component that deals with downloading files asynchronously. When the file is downloaded I then need to process the entire file inside the browser. The wasm module handles the process however wasm has limited memory, so files as large as 5GB must remain in javascript's memory space (or where ever downloaded file data is stored, I'm not sure). In order to process the data -- for example, the aforementioned 5GB -- I must stream it through the wasm module.
To do this I

  1. Send a XMLHttpRequest with `responseType = blob` that will in turn GET a large file
  2. Once downloaded (`ReadyState==4`), get the XMLHttpRequest's `response` object - which will be a blob - then call it's [`stream()`](https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream).
  3. With that stream, call `getReader()` (with no arguments) which will return a `ReadableStreamDefaultReader` (note I would like to use a BYOB buffer [however its not supported at all at this moment.](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamBYOBReader#Browser_compatibility))
  4. With that `ReadableStreamDefaultReader`, all calls to [`read`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader/read) yields a chunk that is always 0x10000 in length, the chunk can be smaller, but only if it is the last amount of data being returned by the file.

Now, none of this is a problem (except the fact that I am [forced to use recursive-async engineering for a simple read command](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader/read#Example_1_-_simple_example)). However in step 4., I'm assuming that my wasm module will only ever need a buffer that's 0x10000 bytes in length. That number is not specified anywhere. It would be handy if it was... then I -- as a WASM developer -- would know exactly how much memory I need to allocate for all applications, making my wasm very efficient 




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/w3c/FileAPI/issues/144#issuecomment-570818483