Re: [mediacapture-transform] Is MediaStreamTrackProcessor for audio necessary? (#29)

It was mentioned in this thread that a `MediaStreamTrackProcessor` for audio is necessary to synchronize audio and video when using WebCodecs.

But in case I didn't miss anything it's probably still hard to accurately encode a `MediaStream` with WebCodecs even though there is `MediaStreamTrackProcessor` for audio.

I tried to record the `MediaStream` coming from the user's mic and camera in Chrome v105. It was obtained in the most simple way.

```js
const mediaStream = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
})
```

I then used a `MediaStreamTrackProcessor` for each `MediaStreamTrack` to get the `AudioData` and `VideoFrame` respectively. However the timestamp of the video seems to start at 0 whereas the timestamp of the audio starts somewhere. 

I think this is all fine according to the spec but it doesn't really help to synchronize the audio with the video. If I want to start the recording at a given point in time which `VideoFrame` and which `AudioData` are the first ones I should pass on to the encoder?

It would be nice to have a way of knowing the offset between the two timestamps. I think some API which says `AudioData.timestamp === 62169.819898` and `VideoFrame.timestamp === 0.566633` represent the same point in time would be really helpful.

Also I guess this all becomes very tricky when the recording is long enough for the two streams to drift apart.

-- 
GitHub Notification of comment by chrisguttandin
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/29#issuecomment-1236609041 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 5 September 2022 06:57:03 UTC