W3C home > Mailing lists > Public > public-webrtc-logs@w3.org > April 2019

Re: [mediacapture-record] Proposal: Specify ability to pause and resume between adding and removing MediaStreamTracks to an active MediaStream (#147)

From: guest271314 via GitHub <sysbot+gh@w3.org>
Date: Sun, 07 Apr 2019 11:04:50 +0000
To: public-webrtc-logs@w3.org
Message-ID: <issue_comment.created-480580164-1554635089-sysbot+gh@w3.org>
@yellowdoge cc @Pehrsons Reading the relevant specifications 


[Media Capture from DOM Elements](https://w3c.github.io/mediacapture-fromelement/)

> [3](https://w3c.github.io/mediacapture-fromelement/#html-media-element-media-capture-extensions). HTML Media Element Media Capture Extensions
> 
> Both MediaStream and HTMLMediaElement expose the concept of a track. Since there is no common type used for HTMLMediaElement, **this document uses the term track to refer to either VideoTrack or AudioTrack**. MediaStreamTrack is used to identify the media in a MediaStream.

[HTML Standard](https://html.spec.whatwg.org/multipage/media.html)

> [4.8.12.10](https://html.spec.whatwg.org/multipage/media.html#media-resources-with-multiple-media-tracks) Media resources with multiple media tracks
> 
> **A media resource can have multiple embedded audio and video tracks.** For example, in addition to the primary video and audio tracks, a media resource could have foreign-language dubbed dialogues, director's commentaries, audio descriptions, alternative angles, or sign-language overlays.
>
> **There are only ever one AudioTrackList object and one VideoTrackList object per media element, even if another media resource is loaded into the element: the objects are reused. (The AudioTrack and VideoTrack objects are not, though.)**

[WebRTC 1.0: Real-time Communication Between Browsers](http://w3c.github.io/webrtc-pc)

[5.2](http://w3c.github.io/webrtc-pc/#rtcrtpsender-interface) RTCRtpSender Interface

[`replaceTrack`](http://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-replacetrack)

> 6.4.3. If sending is true, and withTrack is not null, **have the sender switch seamlessly to transmitting withTrack instead of the sender's existing track.** (emphasis added)
>
>NOTE
>
> Changing dimensions and/or frame rates might not require negotiation. Cases that may require negotiation include:
> 
> 1. Changing a resolution to a value outside of the negotiated imageattr bounds, as described in [RFC6236].
> 2. Changing a frame rate to a value that causes the block rate for the codec to be exceeded.
> 3. A video track differing in raw vs. pre-encoded format.
> 4. An audio track having a different number of channels.
> 5. Sources that also encode (typically hardware encoders) might be unable to produce the negotiated codec; similarly, software sources might not implement the codec that was negotiated for an encoding source.

(issues re `replaceTrack`)

- https://github.com/w3c/webrtc-pc/issues/1677
- https://github.com/w3c/webrtc-pc/issues/2024

[MediaStream Recording](https://w3c.github.io/mediacapture-record)

> [2.3.](https://w3c.github.io/mediacapture-record/#mediarecorder-methods) Methods
> 
> `start(optional unsigned long timeslice)`
> 
> 5. If at any point, a track is added to or removed from the stream's track set, the UA MUST immediately stop gathering data, discard any data that it has gathered, and queue a task, using the DOM manipulation task source, that runs the following steps:
> 
>     1 Set state to inactive.
>     2 Fire an error event named InvalidModificationError at target.
>     3 Fire a blob event named dataavailable at target with blob.
>     4 Fire an event named stop at target.

(issues re `MediaRecorder` and multiple video tracks)

- https://github.com/w3c/mediacapture-record/issues/4
- [Issue 528523: MediaRecorder: Support multiple Video Track recording](https://bugs.chromium.org/p/chromium/issues/detail?id=528523)
- [Issue 1352243002: Implemented Multiple video track recoding.](https://codereview.chromium.org/1352243002/)
- [Issue 894556: Multiple video tracks in a MediaStream are not reflected on the videoTracks object on the video element](https://bugs.chromium.org/p/chromium/issues/detail?id=894556)

et al., the language in MediaStream Recording does not specifically state that a video track cannot be _replaced_ by another video track. To that end [`RTCRtpSender.replaceTrack()`](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/replaceTrack) 

> The RTCRtpSender method replaceTrack() replaces the track currently being used as the sender's source with a new MediaStreamTrack. The new track must be of the same media kind (audio, video, etc) and switching the track should not require negotiation.

can be used to "seamlessly" replace a video track having the same codecs and constraints. 

This issue should be construed as a request for an enhancement of `MediaRecorder` to add a `replaceTrack` method having the same, similar or enhanced functionality of `RTCRtpSender.replaceTrack()` (without having to explicitly use `PeerConnections`; e.g., `recorderInstance.replaceTrack(withTrack)`) which should make it possible (given the same codecs and constraints; or even different codecs and constraints) to replace the current video and/or audio track with a new track. 

Composed a proof of concept using two WebRTC `PeerConnection()`s, which should not be necessary once the method is added to the `MediaRecorder` object, to wit in code


```
<!DOCTYPE html>
<html>

<head>
  <title>Record media fragments to single webm video using AudioCAudioContext.createMediaStreamDestination(), canvas.captureStream(), PeerConnection(), RTCRtpSender.replaceTrack(), MediaRecorder()</title>
  
  <!--
  Try to achieve requirement using only native browser API, without any libraries
  <script src="https://cdnjs.cloudflare.com/ajax/libs/webrtc-adapter/6.4.0/adapter.min.js"></script>
  -->
  <!-- 
    Without using adapter.js at Chromium if {once: true} is not used at "icecandidate" event

    Uncaught (in promise) TypeError: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Candidate missing values for both sdpMid and sdpMLineIndex
    at RTCPeerConnection.
    Uncaught (in promise) TypeError: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Candidate missing values for both sdpMid and sdpMLineIndex
    at RTCPeerConnection.
  -->
  <!-- Without using adapter.js at Firefox

    SecurityError: The operation is insecure. debugger eval code:152
    mediaStreamTrackPromise debugger eval code:152
    dispatchEvent resource://gre/modules/media/PeerConnection.jsm:707
    _processTrackAdditionsAndRemovals resource://gre/modules/media/PeerConnection.jsm:1324
    onSetRemoteDescriptionSuccess resource://gre/modules/media/PeerConnection.jsm:1661
    haveSetRemote resource://gre/modules/media/PeerConnection.jsm:1032
    haveSetRemote resource://gre/modules/media/PeerConnection.jsm:1029
    AsyncFunctionNext self-hosted:839

    at `resolve()`

   -->
  <!-- 
    With using adapter.js at Firefox even with {once: true} set at "icecandidate" event

    from 
    icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:114:21
    from 
    icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
    Tc5OsSypnNbxzidJ:114:21
    from 
    icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:114:21
    from 
icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:114:21
    from 
    icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:114:21
    from 
    icecandidate { target: RTCPeerConnection, isTrusted: true, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, composed: false, … }
Tc5OsSypnNbxzidJ:114:21
    to 
    icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:123:21
    to 
icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:123:21
    to 
    icecandidate { target: RTCPeerConnection, isTrusted: true, candidate: RTCIceCandidate, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, … }
Tc5OsSypnNbxzidJ:123:21
    to 
    icecandidate { target: RTCPeerConnection, isTrusted: true, srcElement: RTCPeerConnection, currentTarget: RTCPeerConnection, eventPhase: 2, bubbles: false, cancelable: false, returnValue: true, defaultPrevented: false, composed: false, … }
Tc5OsSypnNbxzidJ:123:21
  -->

</head>

<body>
  <h1 id="click">click</h1>
  <video id="video" src="" controls="true" autoplay="true"></video>
  <video id="playlist" src="" controls="true" muted="true"></video>
  <script>
    const captureStream = mediaElement =>
      !!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();

    const width = 320;
    const height = 240;
    const videoConstraints = {
      frameRate: 60,
      resizeMode: "crop-and-scale",
      width,
      height
    };
    const blobURLS = [];
    const urls = Promise.all([{
      src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
      from: 0,
      to: 4
    }, {
      from: 10,
      to: 20,
      src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
    }, {
      from: 55,
      to: 60,
      src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
    }, {
      from: 0,
      to: 6,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
    }].map(async({
      from,
      to,
      src
    }) => {
      try {
        const request = await fetch(src);
        const blob = await request.blob();
        const blobURL = URL.createObjectURL(blob);
        blobURLS.push(blobURL);
        return `${blobURL}#t=${from},${to}`;
      } catch (e) {
        throw e;;
      }
    }));


    const playlist = document.getElementById("playlist");
    playlist.width = width;
    playlist.height = height;

    const video = document.getElementById("video");
    video.width = width;
    video.height = height;

    const canvas = document.createElement("canvas");
    const ctx = canvas.getContext("2d");
    canvas.width = width;
    canvas.height = height;

    let recorder;
    let resolveResult;
    const promiseResult = new Promise(resolve => resolveResult = resolve);

    document.getElementById("click")
      .onclick = e =>
      (async() => {
        try {
          // create MediaStream, audio and video MediaStreamTrack
          const audioContext = new AudioContext();
          const audioContextDestination = audioContext.createMediaStreamDestination();
          let mediaStream = audioContextDestination.stream;
          const [audioTrack] = mediaStream.getAudioTracks();
          const [videoTrack] = canvas.captureStream().getVideoTracks();
          // apply same constraints
          videoTrack.applyConstraints(videoConstraints);
          mediaStream.addTrack(videoTrack);

          console.log("initial MediaStream, audio and video MediaStreamTracks", mediaStream, mediaStream.getTracks());

          let tracks = 0;

          const fromLocalPeerConnection = new RTCPeerConnection();
          const toLocalPeerConnection = new RTCPeerConnection();

          fromLocalPeerConnection.addEventListener("icecandidate", async e => {
            console.log("from", e);
            try {
              await toLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
            } catch (e) {
              console.error(e);
            }
          }, {
            once: true
          });

          toLocalPeerConnection.addEventListener("icecandidate", async e => {
            console.log("to", e);
            try {
              await fromLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
            } catch (e) {
              console.error(e);
            }
          }, {
            once: true
          });
          fromLocalPeerConnection.addEventListener("negotiationneeded", e => {
            console.log(e);
          });
          toLocalPeerConnection.addEventListener("negotiationneeded", e => {
            console.log(e);
          });
          const mediaStreamTrackPromise = new Promise(resolve => {
            toLocalPeerConnection.addEventListener("track", track => {
              console.log("track event", track);
              const {
                streams: [stream]
              } = track;
              console.log(tracks, stream.getTracks().length);
              // Wait for both "track" events
              if (typeof tracks === "number" && ++tracks === 2) {
                console.log(stream);
                // Reassign stream to initial MediaStream reference;
                // have only been able so far to get this working 
                // within "track" event referencing the stream property of the event
                mediaStream = stream;
                // set video srcObject to reassigned MediaStream
                video.srcObject = mediaStream;
                tracks = void 0;
                let result;
                recorder = new MediaRecorder(stream, {
                  mimeType: "video/webm;codecs=vp8,opus",
                  audioBitsPerSecond: 128000,
                  videoBitsPerSecond: 2500000
                });
                recorder.addEventListener("start", e => {
                  console.log(e);
                });
                recorder.addEventListener("stop", e => {
                  console.log(e);
                  resolveResult(result);
                });
                recorder.addEventListener("dataavailable", e => {
                  console.log(e);
                  result = e.data;
                });
                recorder.start();
                resolve();
              }
            });
          });
          // Add initial audio and video MediaStreamTrack to PeerConnection, pass initial MediaStream
          const audioSender = fromLocalPeerConnection.addTrack(audioTrack, mediaStream);
          const videoSender = fromLocalPeerConnection.addTrack(videoTrack, mediaStream);
          const offer = await fromLocalPeerConnection.createOffer();
          await toLocalPeerConnection.setRemoteDescription(offer);
          await fromLocalPeerConnection.setLocalDescription(toLocalPeerConnection.remoteDescription);
          const answer = await toLocalPeerConnection.createAnswer();
          await fromLocalPeerConnection.setRemoteDescription(answer);
          await toLocalPeerConnection.setLocalDescription(fromLocalPeerConnection.remoteDescription);
          const media = await urls;
          await mediaStreamTrackPromise;

          console.log(audioSender, videoSender, mediaStream);

          for (const blobURL of media) {
            await new Promise(async resolve => {
              playlist.addEventListener("canplay", async e => {
                console.log(e);
                await playlist.play();
                const stream = captureStream(playlist);
                const [playlistVideoTrack] = stream.getVideoTracks();
                const [playlistAudioTrack] = stream.getAudioTracks();
                // Apply same constraints on each video MediaStreamTrack
                playlistVideoTrack.applyConstraints(videoConstraints);
                // Replace audio and video MediaStreamTrack with a new media resource
                await videoSender.replaceTrack(playlistVideoTrack);
                await audioSender.replaceTrack(playlistAudioTrack);

                console.log(recorder.state, recorder.stream.getTracks());
              }, {
                once: true
              });

              playlist.addEventListener("pause", async e => {
                // await audioSender.replaceTrack(audioTrack);
                // await videoSender.replaceTrack(videoTrack);
                resolve();
              }, {
                once: true
              });
              playlist.src = blobURL;
            });
          }
          recorder.stop();
          blobURLS.forEach(blobURL => URL.revokeObjectURL(blobURL));
          mediaStream.getTracks().forEach(track => track.stop());
          [audioTrack, videoTrack].forEach(track => track.stop());
          fromLocalPeerConnection.close();
          toLocalPeerConnection.close();
          return await promiseResult;
        } catch (e) {
          throw e;
        }
      })()
      .then(blob => {
        console.log(blob);
        video.remove();
        playlist.remove();
        const videoStream = document.createElement("video");
        videoStream.width = width;
        videoStream.height = height;
        videoStream.controls = true;
        document.body.appendChild(videoStream);
        videoStream.src = URL.createObjectURL(blob);
      })
      .catch(console.error);
  </script>
</body>

</html>
```


-- 
GitHub Notification of comment by guest271314
Please view or discuss this issue at https://github.com/w3c/mediacapture-record/issues/147#issuecomment-480580164 using your GitHub account
Received on Sunday, 7 April 2019 11:04:55 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 4 June 2019 15:32:56 UTC