[mediacapture-transform] MediaStreamTrackGenerator/MediaStream should buffer the current frame (#114)

jamiebuilds has just created a new issue for https://github.com/w3c/mediacapture-transform:

== MediaStreamTrackGenerator/MediaStream should buffer the current frame ==
Right now, at least in [Chromium's implementation which they say is to spec](https://issues.chromium.org/issues/370546371):

If you write a frame to `MediaStreamTrackGenerator` and _then_ assign its container MediaStream to a `video.srcObject` you will never see that frame.

```ts
let track = new MediaStreamTrackGenerator({ kind: "video" });
let stream = new MediaStream([track]);
let writer = track.writable.getWriter();

// First write frame:
await writer.write(videoFrame);
// Then assign:
videoElement.srcObject = stream

// Result: Frame never appears
```

This is different than the behavior of a MediaStream created by `HTMLCanvasElement.captureStream()`

```ts
let stream = canvas.captureStream(0) // manually request frames
let track = stream.getVideoTracks().at(0)

// First write frame:
track.requestFrame();
// Then assign:
videoElement.srcObject = stream

// Result: Frame appears
```

Besides the behavior being different, it also makes using these streams inconvenient for using across multiple `<video>` elements which is common in a lot of video calling apps, and you end up needing to hold onto the current frame anyways.

```ts
// This will cause frames to be dropped every time you switch views/speakers/etc:
function onInAppPictureInPicture() {
  let video = document.createElement("video")
  video.srcObject = getCurrentSpeakerStream()

 function onCurrentSpeakerChange() {
   video.srcObject = getCurrentSpeakerStream()
 }

  // ...
}
```

Instead, if you want to avoid recreating the same stream again, you would have to hold onto the last frame and write it to the stream again when necessary:

```ts
let track = new MediaStreamTrackGenerator({ kind: "video" })
let writer = this.#trackGenerator.writable.getWriter()
let lastFrame

async function writeFrame(frame) {
  lastFrame = frame
  await writer.write(frame)
}

async function replayLastFrame() {
  await writer.write(lastFrame)
}
```

This difference between `CanvasCaptureMediaStreamTrack` vs `MediaStreamTrackGenerator` caused a real regression in our app when attempting to use it.

Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/114 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 29 October 2024 17:40:13 UTC