Re: [whatwg/fetch] Getting all bytes in a body (#661)

> What about if it were done using a magic "wait" primitive which was only available "in parallel" (i.e. on background threads)? Well, still a pretty big can of worms, given you'd essentially be talking about manipulating the stream JS objects in parallel :-/. Yeah, let's not do that, at least not until I've really psyched myself up into believing it's clearly the best path forward...

I hadn't thought about that aspect. There needs to be a clear separation between parts that are notionally taking place on some other thread and those parts that may be calling into JS. But it naively appears to be doable with the right wording; something like "await with Javascript context Foo ...".

> Where it lives is less concerning to me than the model. Do we try to emulate Web IDL's style by treating streams as JS objects and then translating them into spec language? E.g. wrapping promises up into "spec async", or Uint8Arrays into Infra byte sequences. Or, do we try to say that there is some "byte stream" or "byte token sequences" concept that is more fundamental, and the JS API lives on top of?

What's the cost of getting it wrong? My inclination is just to start with the text from the Fetch standard and expand on that. But obviously we shouldn't go down a dead-end if it's going to be hard to get back out.

> However I'm also worried about the byte sequences/token sequences confusion I discovered, where lots of other specs treat Encoding's algorithms as if they operate on static byte sequences, whereas Encoding defines them as operating on streams/token sequences that can change over time.

I think they are probably just missing the:

> 3. If input is given, then push a copy of input to stream.

step from [decode()](https://encoding.spec.whatwg.org/#dom-textdecoder-decode) and the:

> 3. If result is finished, convert output into a byte sequence, and then return a Uint8Array object wrapping an ArrayBuffer containing output.

step from [encode()](https://encoding.spec.whatwg.org/#dom-textencoder-encode). Since I don't find it attractive to export the Encoding Standard's "stream" concept, I think it would be preferable to add a couple of algorithmic wrappers around the [run](https://encoding.spec.whatwg.org/#concept-encoding-run) algorithm for use by other standards.

> I think if we had more explicit static byte-sequence-taking endpoints in Encoding, it'd be clearer how much of encoding really needs "streaming" (i.e. token sequences) and thus how much of Encoding needs asynchronicity and all the attendant complexity.

Everything in Encoding is synchronous, so there's no problem there. The kind of asynchronocity I need is to have an implicit conversion from a synchronous algorithm to an asynchronous algorithm so that the algorithms for transform stream support can just say "return" and the Streams standard will see a Promise resolved with undefined.

Encoding's concept of a "stream" is technically speaking a double-ended queue (deque). It's almost a simple queue; It only needs to be double-ended to support the "prepend" operation. No "streaming" as such is required, and I don't think browsers actually implement it this way, it just makes the algorithms clearer.

There's also the confusingly-named "stream" option to decode(), which doesn't "need" streaming but enables callers to implement it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/fetch/issues/661#issuecomment-360066471