[mediacapture-fromelement] Make requestFrame() observable as a Promise resolving with frame metadata (#86)

guest271314 has just created a new issue for https://github.com/w3c/mediacapture-fromelement:

== Make requestFrame() observable as a Promise resolving with frame metadata ==
Currently `requestFrame()` is not observable. 

At some implementations of `captureStream()` it might be necessary to execute `requestFrame()` twice to get the current frame drawn onto the canvas captured in the `MediaStreamTrack`, see https://bugs.chromium.org/p/chromium/issues/detail?id=1047984.

Given `requestFrame()` operation necessarily asynchronous make `requestFrame()` a `Promise` that resolves with a plain JavaScript object containing metadata about the frame request and image, for example, from [HTMLVideoElement.requestVideoFrameCallback()](https://wicg.github.io/video-rvfc/) we find this language

> Drawing operations (e.g. drawing a video frame to a canvas via drawImage()) made through this API will be synchronized as a _best effort_ with the video playing on screen. _Best effort_ in this case means that, even with a normal work load, a callback can occasionally be fired one v-sync late, relative to when the new video frame was presented. This means that drawing operations might occasionally appear on screen one v-sync after the video frame does. Additionally, if there is a heavy load on the main thread, we might not get a callback for every frame (as measured by a discontinuity in the presentedFrames).

where _best effort_ is a realistic description of drawing frames onto the `<canvas>` that will make it to the `MediaStreamTrack`, as there is no guarantee that `requestFrame()` will be executed in the current "event loop" or "tick" or "v-sync".

[2. VideoFrameMetadata](https://wicg.github.io/video-rvfc/#video-frame-metadata)

```
dictionary VideoFrameMetadata {
  required DOMHighResTimeStamp presentationTime;
  required DOMHighResTimeStamp expectedDisplayTime;

  required unsigned long width;
  required unsigned long height;
  required double mediaTime;

  required unsigned long presentedFrames;
  double processingDuration;

  DOMHighResTimeStamp captureTime;
  DOMHighResTimeStamp receiveTime;
  unsigned long rtpTimestamp;
};
```

and 

[3. VideoFrameRequestCallback](https://wicg.github.io/video-rvfc/#video-frame-request-callback)

```
callback VideoFrameRequestCallback = void(DOMHighResTimeStamp now, VideoFrameMetadata metadata);
```

so that users may gain some observable metadata about the result of the transactional procedure of drawing an image onto the `<canvas>` and capturing that image into a `MediaStreamTrack`. 

The resulting change could look something like

```
await track.requestFrame()
/*
[
  3104.6,
  {
    "expectedDisplayTime": 3089.1200000000003,
    "height": 150,
    "mediaTime": 2.805395,
    "presentationTime": 3089.1200000000003,
    "presentedFrames": 3,
    "width": 300
  }
]
*/
```

Please view or discuss this issue at https://github.com/w3c/mediacapture-fromelement/issues/86 using your GitHub account

Received on Wednesday, 1 July 2020 19:55:05 UTC