[whatwg/streams] Allow other specifications to create readable byte streams (#1121)

I'd like to solve https://github.com/WICG/serial/issues/127 and https://github.com/whatwg/fetch/issues/267. /cc @reillyeon @yutakahirano. This issue is trying to figure out how to do that.

Note that Web Serial has a proper pull algorithm and so is very similar to https://streams.spec.whatwg.org/#example-rbs-pull . Fetch instead is more like https://streams.spec.whatwg.org/#example-rbs-push, except it has backpressure support; it can send a suspend and resume signal.

---

We'd have an algorithm "Set up a readable byte stream" similar to https://streams.spec.whatwg.org/#readablestream-set-up. It takes a mandatory autoAllocateChunkSize. (Although maybe consumer specs would just set that to an implementation-defined value anyway, so perhaps we should have streams do that as a default instead of making it mandatory?).

Then we have two cases:

- Specs like Web Serial which only want to enqueue from the pull algorithm. They will always get a BYOB request they can fill exactly.

- Specs like Fetch which want to enqueue whenever the network has data. They will only sometimes have a BYOB request available.

I think the basic primitives they would use are:

- "enqueue bytes": takes an Infra byte sequence and wraps it into a Uint8Array and enqueues it, ignoring any BYOB requests. Specs "should" avoid doing this whenever there's a current BYOB request.
- "current BYOB request": returns... just the number of bytes requested? I think the actual identity of the ArrayBuffer, as well as the offset, are not interesting. So maybe this is "current bytes requested". It needs to also be able to return null if there is no outstanding BYOB request, in which case the spec should use "enqueue". It will never be null inside a pullAlgorithm.
- "respond to current BYOB request": takes an Infra byte sequence (of max length = current bytes requested) and puts it into the appropriate offset in the BYOB request's view. Then calls the spec-equivalent of `byobRequest.respond(byteSequenceLength)`.

  The way this is written makes it seem like there are necessarily copies (from the byte sequence into the ArrayBufferView), so we'd have to note that more efficient implementations are possible, where the OS directly writes into the ArrayBufferView's backing memory, and the byte sequences are just for spec convenience.

Also, we'd need to update https://streams.spec.whatwg.org/#readablestream-need-more-data and its following definitions to work on byte controllers.

---

In practice I think this would look like the following:

- [Web Serial](https://wicg.github.io/serial/#readable-attribute):
  - The pullAlgorithm consults the "current bytes requested" instead of the desired size.
  - The pullAlgorithm uses "respond to BYOB request" with the bytes it gets, instead of wrapping into a Uint8Array and enqueueing.
- [Fetch](https://fetch.spec.whatwg.org/#concept-http-network-fetch):
  - Says something vague about attempting to use the "current bytes requested" to ask the network for the appropriate amount??
  - If "current bytes requested" is non-null, then:
    - If the (post-content-coding) bytes fit into current bytes requested, then "respond to BYOB request" with them.
    - Otherwise... just enqueue them? Or split up the bytes so that you respond to the BYOB request with the first N and then enqueue the rest?
  - If "current bytes requested" is null, then:
    - Enqueue the bytes
    - Ask the user agent to suspend the ongoing fetch

---

Any thoughts? How does this sound to spec writers who might use it?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/issues/1121