Re: [whatwg/streams] Add a proof of concept of optimized pipe (#511)

Wow, perfect, thank you for the explanation! I understand this is not necessarily something for the spec, but I think it will be valuable to have written up somewhere that we can point people to as an example of how to use the spec's flexibility around piping to accomplish great things. It might be worth prototyping as well, if we have the time and energy; I'm not sure if that was your plan in this PR or if you were just using the .js code as a way to organize the thoughts.

So let me try to restate the actors in the system:

- **pipe requests** represent a pipeTo call, i.e. (readable, writable, options) tuples
- **pipes** represent actual data transfers between readable and writable streams
  - they include information on the transfer method, so they are roughly (readable, writable, transfer method) tuples.
- the **pipe manager** is responsible for finding a series of pipes that satisfy all ongoing pipe requests
  - it probably also is responsible for ensuring that no matter what pipes are in play, the options for each pipe _request_ are honored

With this in mind, the notifyPipeCandidates/waitPipeCandidatesChange mechanisms are all about coordinating to potentially update the set of pipes in action when the set of pipe requests in action change. This is what the reorganizing pipes section is about, I think?

I'm not sure I fully understand the candidate propagation section, or why it implies a new backpressure mechanism. My vision of identity transform streams (which is not fully thought out, so might be wrong) is that in a system like this, they get completely excluded from the list of pipes, i.e. `rs.pipeThrough(id).pipeTo(ws)` will create two pipe requests but only one pipe (from `rs` to `ws`). Then normal backpressure propagation would take place between `rs` and `ws`.

I guess you are concerned about cases like `rs.pipeThrough(id); setTimeout(() => id.readable.pipeTo(ws), 10000)`, when you talk about asynchronocity? Or is it something else?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/pull/511#issuecomment-243924205