Re: [webrtc-pc] Add RTP timestamp to RTCRtpContributingSource (#2177)

I think that a per-frame ID (of some kind) would be very nice to have for `RTCRtpContributingSource`. For example: It would allow a caller to tell if two consecutive `getContributingSources()` calls returned the info about the exact same frame, or if they returned info about two different frames but `timestamp` didn't have enough resolution to tell the frames apart.

There are a bunch of challenges for the spec:

1. A single RTP packet may contain a payload that maps to multiple frame-deliveries to `MediaStreamTrack`.

    For example:
    - Audio packets store 20 ms of audio but WebRTC delivers 10 ms per `GetAudioFrameWithInfo()` call.

2. Multiple RTP packets may be involved in the decoding of a single frame.

    For example:
    - Video key frames.
    - Audio packets that are not aligned with audio outputs. Say we have a packet with audio for time period [1005, 1025) and another for [1025, 1045), but WebRTC is outputing [1020, 1030). It would then take audio samples from both packets.
    - Audio during acceleration. Similar to the case above.

3. Frames that are decoded from packets that don't follow the usual rules about RTP timestamps and sequence numbers.

    For example:
    - Transport-level forward error correction.
      - E.g. Flexfec.
      - We'll presumably use the recovered RTP timestamp.
    - In-band forward error correction.
      - E.g. Opus FEC where WebRTC can use the data from packet N+1 to decode a low-quality version of packet N.
      - Do we use the RTP timestamp for N+1 or the "would have been" RTP timestamp for N?
      - Probably most sense to compute and use the info for N?
      - Or maybe not record anything for this case at all?
    - Retransmission.
      - The retransmitted packet should have the same RTP timestamp as the original.

-- 
GitHub Notification of comment by chxg
Please view or discuss this issue at https://github.com/w3c/webrtc-pc/issues/2177#issuecomment-487033029 using your GitHub account

Received on Friday, 26 April 2019 12:03:40 UTC