[webrtc-pc] Creation of MediaStreamTrack from raw PCM stream; Example 18 with a JSON file instead of messaging (#2570)

guest271314 has just created a new issue for https://github.com/w3c/webrtc-pc:

== Creation of MediaStreamTrack from raw PCM stream; Example 18 with a JSON file instead of messaging ==
To workaround Chromium refusal to support exposure and capture of monitor devices for `getUserMedia()` at Linux have created several workarounds https://github.com/guest271314/captureSystemAudio/ that, from perspective here, could be simplified.

Am able to get a `ReadableStream` where the value when read is raw PCM audio data, 2 channel, 44100 sample rate, (s16le).

The current version uses browser extension and Native Messaging to `fetch()` from localhost where output is the monitor device data passed through `PHP` `passthru()`, then due to Chromium extension messaging code not supporting transfer, converted to text, then messaged to a different origin.

There has to be a simpler way to do this.

Two options occur, though there are certainly other options that perhaps have not conceived of yet, thus this question

1. Somehow create a `MediaStreamTrack` directly from raw PCM input and somehow get that `MediaStreamTrack` exposed at a different origin
2. Use Native File System to wwrite and read either a single file or multiple files to accomplish ICE, offer, answer

Option 1. is probably more involved, though avoids using `WebAssembly.Memory`, `SharedArrayBuffer`, `ArrayBuffer`, `TypedArray`s, which have limitations both by default design and architechture, https://bugs.chromium.org/p/v8/issues/detail?id=7881#c60

> #59: No, it does not. WebAssembly.Memory.grow() will attempt to grow all the way up to 4GB. On 32-bit systems, it is very unlikely that a large amount of contiguous memory address space is still free, so the kernel will often deny such requests. (There's no hard limit on that; in a fresh process you might be able to allocate as much as 2GB, with sufficient address space fragmentation in a long-lived process even getting 256MB might fail.)
> 
> Unrelated to memory growth itself, the limit for TypedArray length on 32-bit systems is currently (and will likely continue to be) 1GB. So while you might be able to grow a Wasm memory to be bigger than that, you won't be able to create a TypedArray spanning its entire underlying ArrayBuffer.

That limitation is observable at 32-bit systems https://github.com/wasmerio/php-ext-wasm/issues/121#issuecomment-678898047 where when attempting to dynamically use `WebAssembly.Memory.grow(1)` when the current `value` (`Uint8Array`) plus previously written values exceed initial or current `SharedArrayBuffer` `byteLength` ultimately allocation of increased memory can not succeed; e.g., trying to capture 30 minutes of audio (which is written to memory, while memory grows dynamically) can result in only 16 minutes and 43 seconds of audio being recorded. 

Option 2. is probably the simplest in this case. Somehow create a `MediaStreamTrack` from raw PCM input, then write and read offer and answer, and "ICE", negotiation, et al. using a local file (`JSON`) in order to avoid using JavaScript `ArrayBuffer`, `SharedArrayBuffer`, `WebAssembly.Memory.grow(growNPages)` at all.

However, to achieve that am asking the precise flow-chart of exchanges of offer and answer between two `RTCPeerConnection`s, as the instances will be on different origins, and Chromium extension messaging could occasionally require reloading the extension, using Natuive File System to write and read the file(s) bypasses the need to use messaging which requires communication between Native Messsaging host, Chromium extension and arbitrary web page.

Can the specification be updated with an example of performing the _complete_ necessary steps to establish peer connection _for both sides of the connection_, taking Example 18 https://w3c.github.io/webrtc-pc/#example-18 as the base case, using a single JSON file, or multiple files if needed, instead of signaling (messaging)?

In this case the extension code will make to send-only offer and the arbitrary web page will make the receive-only answer.

Ideally, we should be able to somehow just pass the raw PCM to a method of `RTCPeerConnection`, et al. and not use Web Audio API `AudioWorklet` or `TypedArray` or `ArrayBuffer` at all. Is that possible?

Alternatively, are there any other ways to solve this problem that am not considering?



Please view or discuss this issue at https://github.com/w3c/webrtc-pc/issues/2570 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Sunday, 30 August 2020 15:48:08 UTC