Re: [streams] Support reading bytes into buffers allocated by user code on platforms where only async read is available (#253)

@yutakahirano

> (A) ... There is no pulling mode.

Minor point: there is no consumer-initiated pulling, I agree. But a non-byte stream implementation could "pull" ahead of time if it wanted, to make future calls to read() fulfill immediately.

> (B), (C)

I agree with your characterization of these. I had not separately considered (B) before actually.

> (D)

Yeah, I do not like (D) because of the not-a-subtype thing.

> Manual mode ReadableByteStream in (C) and (D) doesn't allow push sources

Speaking about (C): I think it does allow push sources. Based on my understanding of the POSIX APIs underlying what I call "push sources," i.e. epoll + recv, I think it would still work. We would just not call recv until we have been fed a buffer, i.e., we would leave the data in the kernel buffers.

My previous understanding of push sources was that they were more like WebSocket ([see spec example](https://streams.spec.whatwg.org/#example-rs-push-no-backpressure)). I agree that (C) does not work for wrapping WebSocket-type interfaces. So if the APIs we have to wrap are at that level then (C) is no good. But now that I understand the socket APIs better, I think we are OK with something like (C).

> @domenic, is your "async pull + sync read" (B) or (C)?

It was similar to (C). However, I did not envision an auto-mode. Instead, it was always manual-mode, but if you omitted the parameter to pull() it would auto-allocate a buffer for you. I am not sure about the value of auto-mode if we can make manual-mode ergonomic in this way.

I guess it helps in the same way that a non-zero high water mark usually helps: it lets you queue up a few extra chunks that might be useful as a "buffer" (in the other sense of the word) if the source slows down. Hmm. So it is a tradeoff about letting you get more manual control via bring-your-own-buffer (manual mode) vs. automatically queuing up extra chunks for you (auto mode). That is kind of nice as an option to offer, especially since it falls out of the existing design. Maybe it is just something like `.stream()` gives a `ReadableByteStream` and `.stream({ autoFlow: true })` gives a `ReadableStream`.

> I still think (A) is better than (B).

I agree.

---

@tyoshino

> (A) looks reasonable landing point though I miss the sync read() interface we've been developing.

Your reasons for missing it are basically those in https://github.com/whatwg/streams/issues/253#issuecomment-76958542 right?

Maybe it would be helpful for me to spend some time today pretending to be super-opposed to async read and writing up the most strongly-stated arguments I can come up with against it. Similar to yours but more forcefully, and with examples. Then I can switch back to being more rational and evaluate those arguments objectively to see if it helps us come to a conclusion on async read().

> (C) is great ... yes, I understand it's smart, but ... hmm. It made one who doesn't know bring-your-own-buffer reading functionality to call wait() to invoke pull-with-auto-allocate without realizing that he/she is pulling by wait() which he/she called in order to obtain readyPromise ... But, ... hmmm

I think it works better if we rename wait() to pull() like you suggested :).

> I think there's (D)-ii where we use ready + sync read (for auto mode) and async read (for manual mode). (D) is an approach where we give up reduction of usage difference. For manual mode, we can choose any API.

Right, I agree with this characterization. I don't think we should give up on this if we don't have to though.

So to me it comes down to (A) vs. (C). I will try to do my mental exercise of arguing against async read so we can see how strong the arguments are. Maybe that will help.

---

> I'm now convinced and less interested in operation stream. I'll just extract useful proposals from the branch and present to you.

I am still hopeful it will help simplify implementation, probably of readable/writable, and definitely for transform. But yes, it is _definitely_ a good source of proposals.


---
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/issues/253#issuecomment-77197754