Re: [mediacapture-transform] What is the impact of timestamp for video frames enqueued in VideoTrackGenerator? (#96)

VideoTrackGenerator should probably not care about `VideoFrame.timestamp`.
Its processing of frames is to `Send clone to track`, which is bit vague.
My interpretation is that it submits the frame to each one of the track's sink immediately, without any buffering.
`timestamp` has therefore no impact on what VideoTrackGenerator does.
Maybe we should clarify this in the spec.

It is then up to each sink to determine what to do with the VideoFrame timestamp.
RVFC is stating how to populate `captureTime` based on the track type:
`For video frames coming from a local source, this is the time at which the frame was captured by the camera. For video frames coming from remote source, the capture time is based on the RTP timestamp of the frame and estimated using clock synchronization`.

Given track->MSTP->pipeTo->VTG->RVFC should be the same as track->RVFC, we could have the following rules:
1. RFVC is setting the capture time as the value of `VideoFrame.timestamp`.
2. `VideoFrame.timestamp` for camera/screencapture tracks is the capture time as defined in RVFC
3. `VideoFrame.timestamp` for webrtc remote tracks is the capture time as defined in RVFC
4. `VideoFrame.timestamp` is preserved as is by VideoTrackGenerator
5. `VideoFrame.timestamp` for canvas track is the time at which the canvas is snapshotted (similar to camera tracks somehow)
6. Each other sink should describe how it is using `VideoFrame.timestamp`, meaning HTMLVideoElement, RTCRtpSender and MediaRecorder. By default, it does nothing with it.

I think 1, 2, 3 and 4 are aligned with @aboba Chrome's testing but this does not seem described anywhere.
Ideally, each spec defining a video source would define how VideoFrame objects exposed via MSTP are generated, including the timestamp, metadata and so on.

Looking at the sink side in Safari, `VideoFrame.timestamp` is not used much:
- HTMLVideoElement will render the frame ASAP.
- RTCRtpSender and MediaRecorder are roughly using the time at which is submitted the frame as the actual timestamp.
I wonder what Chrome and Firefox are doing for those sinks.
@alvestrand, @jan-ivar, do you know?

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/96#issuecomment-2360878068 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 19 September 2024 12:39:39 UTC