Re: [webrtc-pc] On track removal: Should you mute an already muted track? (#2506)

Attempted to use the same code at Firefox and Chromium to 1) determine what is a minimum requirement for a _remote_ `MediaStreamTrack` from `canvas.captureStream()` to fire `unmute` event; 2) for `HTMLVideoElement` to fire `loadedmetadata` event; 3) for `HTMLVideoElement` to fire `timeupdate` event.

**Observations**

Chromium fires `unmute` irrespective of whether or not a method of the specific context set at the source `canvas` is executed or the canvas is otherwise changed.

Firefox does not fire `unmute` event when no canvas context method is executed and the canvas is not changed.

Two canvas `2d` context method to execute for `unmute` of remote track to fire are `clearRect()` or `fillRect()`. `loadedmetadata` of `HTMLVideoElement` also fires following `unmute` for each browser.

At Firefox `<video>` `timeupdate` is fired and `currentTime` progresses when `autoplay` is set, continues streams content ("black frames"), theoretically, infinitely.

At Chromium `timeupdate` is not fired and `currentTime` does not progress, similar to the behaviour described at https://bugs.chromium.org/p/chromium/issues/detail?id=1045832. The video display is a "black frame". If the play control is pressed `0` is alerted from `timeupdate` event.

Given `unmute` will not be fired at browsers under current specifications and implementations without some value being set or method being called on the remote source canvas  a `RTCDataChannel` can be used to signal the remote peer to perform the required task for the stream to be unmuted. 

That does not solve the Chromium issue of `unmute` being fired on the remote track without any change to thesource  canvas however not progressing `currentTime` or firing `timeupdate` event at `HTMLVideoElement` when a change is made to the canvas.

One option would be to update the canvas capture stream specification to stream the contents of the canvas regardless of whether or not a new image was painted onto or modification was made to the canvas. 

Alternatively, adjust `RTCPeerConnection` to _not_  await _content_ to stream "black frames" from a `<canvas>`, which is already an image, thus already content. This effectively provides a "paused" `MediaStreamTrack`, which could be useful for certain purposes, though the behaviour does not appear to be by design of at least `HTMLVideoElement` or `RTCPeerConnection`, if in fact `unmute` is based on networking, not content.

```
<!DOCTYPE html>

<html>
  <head>
    <title>
      unmute remote RTCPeerConnection from canvas.captureStream(), fire
      HTMLVideoElement loadedmetadata and timeupdate events
    </title>
    <script src="RTCPeerConnection-helper.js"></script>
  </head>

  <body>
    <code>
      unmute remote RTCPeerConnection from canvas.captureStream(), fire
      HTMLVideoElement loadedmetadata and timeupdate
    </code>
    <script>
      (async _ => {
        const v = document.createElement('video');
        v.width = v.height = 200;
        v.controls = v.autoplay = true;
        const pc1 = new RTCPeerConnection();
        const pc2 = new RTCPeerConnection();
        // not fired at Chromium for fillRect or clearRect
        v.ontimeupdate = e => {
          v.ontimeupdate = null;
          alert('currentTime:' + e.target.currentTime);
        };
        const dc2 = pc2.createDataChannel('canvas-stream');
        document.body.appendChild(v);
        const metadataToBeLoaded = new Promise(
          resolve => (v.onloadedmetadata = e => resolve(e.type))
        );
        const canvasStream = () => {
          const canvas = document.createElement('canvas');
          canvas.width = canvas.height = v.width;
          const ctx = canvas.getContext('2d');
          const stream = canvas.captureStream();
          const [track] = stream.getVideoTracks();
          const dc1 = pc1.createDataChannel('canvas-stream');
          dc1.onopen = e => {
            dc1.onmessage = e => {
              ctx.clearRect(0, 0, canvas.width, canvas.height);
              dc1.close();
            // do stuff
            };
          };
          return stream;
        };

        let stream = canvasStream();
        let [track] = stream.getVideoTracks();
        const sender = pc1.addTrack(track);

        pc2.ontrack = async e => {
          e.track.onunmute = e => {
            alert(e.type);
            v.srcObject = new MediaStream([e.target]);
          };
          pc2.ondatachannel = e => {
            e.channel.send(null);
            e.channel.close();
          };
        };

        exchangeIceCandidates(pc1, pc2);
        doSignalingHandshake(pc1, pc2);
        let result = await metadataToBeLoaded;
        alert(result);
        // pc1.close();
        // pc2.close();
      })();
    </script>
  </body>
</html>
```

plnkr https://plnkr.co/edit/9XcS0fbCPfgvPBzd?preview

-- 
GitHub Notification of comment by guest271314
Please view or discuss this issue at https://github.com/w3c/webrtc-pc/issues/2506#issuecomment-618206122 using your GitHub account

Received on Thursday, 23 April 2020 06:29:36 UTC