W3C home > Mailing lists > Public > public-webrtc-logs@w3.org > July 2019

Re: [mediacapture-main] Remove "The relative order of the tracks in the set is User Agent defined and the API will never put any requirements on the order.", et al. (#611)

From: guest271314 via GitHub <sysbot+gh@w3.org>
Date: Mon, 22 Jul 2019 23:52:11 +0000
To: public-webrtc-logs@w3.org
Message-ID: <issue_comment.created-513997430-1563839530-sysbot+gh@w3.org>
@youennf Own interest in the track order stems from the yet unmet task of porting `mkvmerge` to JavaScript. In lieu of a full port of the program written by hand and/or using Emscripten and/or `WebAssembly`, composed code that uses Native Messaging and Native File System in order to effectively use `mkvmerge` in the browser. While doing so encountered several issues with how the WebM files of browsers are output. When arbitrary track order exists, it is necessary to write code to determine the current and previous tracks. AFAIK there is no standardized means to do so, though there is an algorithm that can be adhered to [mkvmerge prints error when merging webm files from MediaRecorder at Chromium and MediaRecorder at Firefox. Why?](https://www.reddit.com/r/mkvtoolnix/comments/cdi824/mkvmerge_prints_error_when_merging_webm_files/etwhugs/). To merge two files there are only two choices. To merge four, twenty or a hundred files the algorithm _should_ be the same, though since there is no standardized roadmap of how to check that the map for appending is correct for all cases 

https://github.com/guest271314/native-messaging-mkvmerge/blob/master/app/native-messaging-mkvmerge.js#L187

```
    // https://www.reddit.com/r/mkvtoolnix/comments/cdi824/mkvmerge_prints_error_when_merging_webm_files/etwhugs/
    // 1. run `mkvmerge -J` file.webm on each input file
    // 2. determine which tracks go together
    // 3. create an appropriate --append-to argument from the algorithm in 2
    let filesMetadata = [];
    for (const fileName of fileNames) {
       await new Promise(resolve => {
         const getMetadata = ({body}) => {        
           port.onMessage.removeListener(getMetadata);
           filesMetadata.push(JSON.parse(body));
           resolve();
         };
         port.onMessage.addListener(getMetadata);
         port.postMessage({
           "message": "metadata",
           "body": `${cmd} ${metadata} ${fileName}`
         });
       });
    };
    // construct `--append-to` option for merging files where
    // tracks are not in consistent order; for example, WebM
    // files output by Chromium, Firefox MediaRecorder implementations 
    // Chromium => Opus: "id": 0, Firefox => Opus: "id": 1,
    // Chromium => VP8: "id": 1, Firefox => VP8: "id": 0 
    for (let i = 0; i < filesMetadata.length; i++) {
      const {tracks:currentTracks} = filesMetadata[i];
      const currentAudioTrack = getTrack(currentTracks, "audio").id;
      const currentVideoTrack = getTrack(currentTracks, "video").id;
      if (filesMetadata[i + 1]) {
        const {tracks:nextTracks} = filesMetadata[i + 1];
        const nextAudioTrack = getTrack(nextTracks, "audio").id;
        const nextVideoTrack = getTrack(nextTracks, "video").id;
        appendTo += `${i+1}:${nextAudioTrack}:${i}:${currentAudioTrack},${i+1}:${nextVideoTrack}:${i}:${currentVideoTrack},`;
      } 
      else {
        const {tracks:previousTracks} = filesMetadata[i - 1];
        const previousAudioTrack = getTrack(previousTracks, "audio").id;
        const previousVideoTrack = getTrack(previousTracks, "video").id;
        appendTo += `${i}:${currentAudioTrack}:${i-1}:${previousAudioTrack},${i}:${currentVideoTrack}:${i-1}:${previousVideoTrack}`;
      }
    };   
    // check if tracks are ordered AV,AV...AV or arbitrarily AV,VA,AV,AV,VA...AV
    const orderedTracks = filesMetadata.map(({tracks}) => tracks).every(([{type}]) => type === "audio");
    console.log(JSON.stringify({filesMetadata, orderedTracks, appendTo}, null, 2));
    port.onMessage.addListener(onNativeMessage);
    const message = {
      "message": "write",
      // if tracks in files are not ordered use `--append-to` option, else do not
      "body": `${cmd} ${options} ${outputFileName} ${!orderedTracks ? appendTo : ""} '[' ${fileNames.join(" ")} ']'`
    };
```
How to determine that the implementation of the algorithm is correct for any conceivable ordering tracks  of _N_ WebM files created by Chromium/Chrome, Firefox and Safari?

Would prefer to not have to ask the author of `mkvmerge` if the `--append-to` mapping is correct for every merging of files over two, for example, for a total of 10 files created at both Firefox and Chromium there could be various ways the order of the resulting map needs to be arranged - and, though outside the scope of this issue, then it must be determined if the audio have `1` or more channels, and adjust the audio tracks to be consistent among all files to be merged.

If the ordering of tracks was specified, and adhered to, that would be one less item to be concerned with when merging WebM (or Matroska) files created by using `MediaStream` and `MediaRecorder`. If the `audio_sampling_frequency` was also specified to be consistent among implementers, then we would have a semblance of consistency, or "interoperablity" at the front-end. Specifying tracks be ordered in a consistent, agreed upon sequence should not break the web at all; in fact such specificity would improve interoperabilty and provide an web environment where the web will be less likely to break, that is, if implementers stay in accord with the specification, as at least one Firefox developer apparently contends Chrome authors did not even though Chrome (*oogle) initially pushed for WebM for, ostensibly, similar reason.

-- 
GitHub Notification of comment by guest271314
Please view or discuss this issue at https://github.com/w3c/mediacapture-main/issues/611#issuecomment-513997430 using your GitHub account
Received on Monday, 22 July 2019 23:52:13 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:22:25 UTC