Re: [mediacapture-image] onframe Event (#210)

I made a rough implementation of this in Chromium for my own testing purposes, and quickly realized he importance of ensuring the events don't tie up too much memory. Here's a proposed change to the spec to accomplish that:

```webidl
partial interface ImageCapture {
  attribute EventHandler onframe;
}

[Exposed=Window]
interface ImageCaptureFrameEvent : Event {
   Promise<ImageBitmap> createImageBitmap();
};
```

I left `ImageCaptureFrameEvent` barebones for the moment, but there are several potential attributes which might be useful to include, which would make the event also a good passive way to collect info about the `MediaStreamTrack`.

The promise for `createImageBitmap` will reject if the browser has already discarded the memory associated with that frame. This makes it efficient to listen to frame events passively without causing lots of garbage collection or high memory usage.

There's a lot of benefit to having a `frame` event. It ensures that you can be notified of every frame delivered by the `MediaStreamTrack`, makes it easy to tell when you're getting behind in processing, and provide flexibility in how to handle those frames. The current implementation of `grabFrame` in Chromium will not resolve until the next frame available, which adds undue latency at low framerates if you just want a current shot of the `MediaStreamTrack`, and requires unnecessary CPU usage if you want to process every other frame since it requires creating an `ImageBitmap` for the frames you plan on skipping anyway. Trying to use wall time to accomplish this sort of behavior is fragile and more complicated than it should be.

Below is an example of how to run `grabFrame` only up to the `MediaStreamTrack` frame rate, using the proposed `frame` event. Should highlight the flexibility of this approach, 

```javascript
class ThrottledImageCapture {
  constructor (capturer) {
    this.bufferCount_ = 2;
    this.buffers_ = [];

    this.dropCount_ = 0;
    this.discardCount_ = 0;

    this.pendingGrabFramePromise_ = null;

    capturer.addEventListener('frame', async (event) => {
      if (this.buffers_.length === this.bufferCount_) {
        // We ran out of buffers, so we had to
        // drop a frame due to being too slow
        const frame = this.buffers_.shift();
        frame.close();  // Dispose of memory
        this.dropCount_++;
      }

      try {
        this.buffers_.push(await event.createImageBitmap());
      } catch {
        // JS execution was too slow and the browser
        // discarded the data for the frame before we
        // could get it as an ImageBitmap
        this.discardCount_++;
        return;
      }

      if (this.pendingGrabFramePromise_) {
        // Pending promise, consume the latest frame
        this.pendingGrabFramePromise_(this.buffers_.shift());
        this.pendingGrabFramePromise_ = null;
      }
    });
  }

  grabFrame () {
    if (this.buffers_.length) {
      // Frame available, so consume it
      return Promise.resolve(this.buffers_.shift());
    }

    // No frame available, wait for next one. Reuse
    // promise if already created
    if (this.pendingGrabFramePromise_) {
      return this.pendingGrabFramePromise_;
    }

    return new Promise(resolve => {
      this.pendingGrabFramePromise_ = resolve;
    });
  }
}
```

-- 
GitHub Notification of comment by dsanders11
Please view or discuss this issue at https://github.com/w3c/mediacapture-image/issues/210#issuecomment-511709488 using your GitHub account

Received on Tuesday, 16 July 2019 07:58:57 UTC