Re: [webrtc-pc] Add RTP timestamp to RTCRtpContributingSource (#2177)

I think that a per-frame ID (of some kind) would be very nice to have for `RTCRtpContributingSource`. For example: It would allow a caller to tell if two consecutive `getContributingSources()` calls returned the info about the exact same frame, or if they returned info about two different frames but `timestamp` didn't have enough resolution to tell the frames apart.

There are a bunch of challenges for the spec:

1. A single RTP packet may contain a payload that maps to multiple frame-deliveries to `MediaStreamTrack`.

    For example:
    - Audio packets store 20 ms of audio but WebRTC delivers 10 ms per `GetAudioFrameWithInfo()` call.

2. Multiple RTP packets may be involved in the decoding of a single frame.

    For example:
    - Video key frames.
    - Audio packets that are not aligned with audio outputs. Say we have a packet with audio for time period [1005, 1025) and another for [1025, 1045), but WebRTC is outputing [1020, 1030). It would then take audio samples from both packets.
    - Audio during acceleration. Similar to the case above.

3. Frames that are decoded from packets that don't follow the usual rules about RTP timestamps and sequence numbers.

    For example:
    - Transport-level forward error correction.
      - E.g. Flexfec.
      - We'll presumably use the recovered RTP timestamp.
    - In-band forward error correction.
      - E.g. Opus FEC where WebRTC can use the data from packet N+1 to decode a low-quality version of packet N.
      - Do we use the RTP timestamp for N+1 or the "would have been" RTP timestamp for N?
      - Probably most sense to compute and use the info for N?
      - Or maybe not record anything for this case at all?
    - Retransmission.
      - The retransmitted packet should have the same RTP timestamp as the original.

GitHub Notification of comment by chxg
Please view or discuss this issue at using your GitHub account

Received on Friday, 26 April 2019 12:03:40 UTC