Re: Raw data API - 3 - Encoded data

This gets to my question I want to cover at the f2f (and which we now have
agenda time for): how low-level an API do we want?  With a low-level RTP
transport object, a JS/wasm app could implement a high-level RtpSender on
top.  But is that the right balance between flexibility and ease of use?
How much do we want to rely on libraries to fill in the higher-level
stuff?

I feel like providing low-level components and letting the js/wasm
libraries fill in the higher-level stuff is the right way to go for most
things, but finding the right line is a bunch of tradeoffs and I think
there's a lot of the WG to consider.

On Tue, May 29, 2018 at 3:24 AM Sergio Garcia Murillo <
sergio.garcia.murillo@gmail.com> wrote:

> I fail to see what would be the benefits of adding the media frame API to
> the RTP objects, especially if we intend to provide a lower raw rtp API.
>
> In order to be able to implement this API on the browser, the codec
> packetization of the encoded stream must be known by the browser. Also, it
> is not possible to modify the frame, for example applying end to end media
> encryption, as packetization requires to have access to the raw data or add
> metadata for the very same reason.
>
> If the API would only alow to forward the frames from an encoder/decoder
> (or another media source/sink) to the rtp objects, I would prefer a higher
> level API that deals with streams and not individual packets.
>
> Best regards
>
> Sergio
>
> On 29/05/2018 8:30, Harald Alvestrand wrote:
>
> *Access to encoded streams*
>
>
>
>
>
>
> * A similar interface can be added to RTPSender and RTPReceiver,
> respectively (and similar APIs for other transports, when defined). Here
> the buffers would contain encoded video / audio data, and the control block
> parameters would have to have enough information to let the RTP packet
> headers be constructed - or, on the receiver, the info from the RTP packet
> headers be represented. Partial interface RTPSender {
>    promise<encodedBuffer> injectData(encodedBuffer); } Partial interface
> RTPReceiver {    promise<encodedBuffer> extractData(encodedBuffer); }
> Interface encodedBuffer : Buffer {    Long rtpTimestamp;    Long frameId;
>    sequence<long> dependsOnFrames;     // more fields TBD }    On this
> interface, frames do have interdependencies, so dropping packets is much
> more problematic. The “dependsOnFrames” member is intended to help deciding
> on sensible handling - it would tell the other side of the API that “if you
> dropped one of these frames, you might as well drop this frame too”. *
>
>
>