Re: [streams] Real time source and skipping (#383)

@wanderview an opportunistic read would work, and is basically what I called "an option for a non pulling read". However as I mentioned, I believe a "caching strategy" option on the stream would be a better approach as the consumer could just configure the desired behavior for the stream without having to implement the logic of pulling and discarding. The main difference is that intermediate values wouldn't have to be queued and can be discarded immediately with the second approach.

@domenic here are some comments . Sorry myself for the delayed answer.

> > on demand, they generate a new frame only when needed

> Can you give an example of such a scenario? Is the idea e.g. reading a movie file from a disk?

An example of a stream that could be "on-demand" is a MediaStream for a screen capture source.
The logic in the screen capture could be to only take a capture of the screen when the consumer actually has a need for a new frame, instead of as fast as possible. The operation can be quite expensive. This is different from a MediaStream attached to a webcam source, which would generate frames no matter what. Note that both streams are "live" streams, and are not tied to a seekable source.

> > Should be consider adding a method on readable streams to probe the size of the queue or behavior of the read?

> I don't really understand how this would solve the issue.

Being able to know if the read operation would pull from the the internal queue of the stream or pull from the source would allow to read from the stream until the latest queued value, without the risk of making a read call that would block until a new value is actually generated by the source. An "opportunistic read" where we ask that the read operation only returns a queued value would work too.

> > I thought about a Transform but I'm having a hard time seeing how that would work:

> It seems like you'd need different transforms for each type of source. This comes back to my question of what use cases an on-demand stream is. It doesn't seem unreasonable to distinguish between webcams and movie files. I agree it would be nice if you didn't have to, though.

In the case of screen sharing and webcam streams, the consumer might not know if the source is on-demand or not, since that would be an internal browser implementation detail. We wouldn't know what type of Transform would work for the source.

> This kind of talk in general makes me unclear whether streams are really a correctly matched solution to your problem. The primitive here is something more like "changing value," which has a single "latest value". Maybe it also optionally has a stream (or something) of notifications that the value has changed. The consumer can then go check the latest value, either based on the notifications (if they are present) or based on polling (which seems like a more likely architecture). See also are readable streams appropriate for my use case?

While I agree that streams are not appropriate for all use cases, the part that confuses me is that the stream makes assumptions about what the consumer intends to do with the values, i.e. take and concatenate all of them. While this is a probably the most common use case, it's not the only one. For a same source / stream, one consumer might want to process all values, while another consumer might want to process only some values based on some external logic. The producer of the stream might not know ahead of time what type of consumers it will have. What you're suggesting is for the producer to provide 2 different APIs to be used by different types of consumers. It feels like with a little bit of work, we could accommodate both types of consumers with a single stream API abstraction.

> > real-time, they generate frames constantly no matter what, e.g. a webcam. These sources would probably use push semantics

> It depends on whether the webcam stream is "recording" or "live". If it is recording, then the whole sequence is valuable, and you don't want to throw away any data, so much of this discussion is moot. If it is live, then I would expect its source to be implemented much more like an on-demand stream, like so:

Again, the "recording" or "live" aspect you're referring to in this case is what the consumer decides to do with the stream. The source and stream implementation should be the same, i.e. it provides the video frames captured by the webcam sensor.

Regarding the pseudo code, just pulling and caching the latest value is a little too simple to illustrate my problem.
The issue really arises when the consumer needs to some work with the value pulled, work that potentially takes longer than the time it takes to generate a new value. Here is some pseudo-code I would like to write:

```
let mediaStreamVideoTrack = ... ; // We got the MediaStreamTrack from somewhere
let stream = createStreamOfFramesFromVideoTrack(mediaStreamVideoTrack);
let reader;

function processFrame(frame) {
  return new Promise((resolve, reject) => {
    // Do some heavy work
  });
}

function processNext () {
  return reader.read().then(processFrame).then(processNext);
}

reader = stream.getReader({highWaterMark: 1, cachingMethod: "keepLast"});
processNext();
```

Without a "caching method", you would have to manually pull like this:

```
let latestFrame;

function processNext () {
  if (!latestFrame || reader.queueSize > 0) {
    reader.read().then((frame) => {
      latestFrame = frame;
      processNext();
    });
  } else {
    let frame = latestFrame;
    latestFrame = null;
    processFrame(latestFrame).then(processNext);
  }
}

reader = stream.getReader();
processNext();
```

The drawback of that "draining" approach, on top of being more complicated for the consumer, is that the stream caches intermediate values. In the case of video frames, this can be memory consuming. If the `processFrame` operation is truly non-blocking, the consumer might be able to schedule regular reads from the stream to drain it, but that would make the consumer logic even more complex.

If you want some backgroud on how this issue came up to be, have a look at the thread called ["Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG"](https://lists.w3.org/Archives/Public/public-media-capture/2015Jul/thread.html#msg68) on the W3C media-capture list.

---
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/issues/383#issuecomment-128143844