Re: Encoders (Re: Getting rid of SDP)

On Mon, Mar 5, 2018 at 11:33 PM Harald Alvestrand <harald@alvestrand.no>
wrote:

> On 03/06/2018 12:10 AM, Peter Thatcher wrote:
>
>
>
> On Mon, Mar 5, 2018 at 3:06 PM Sergio Garcia Murillo <
> sergio.garcia.murillo@gmail.com> wrote:
>
>> More flexibility, more work, more bug surface.. ;)
>>
>> Anyway, I am not particularly against having access to RTP packets from
>> the encoders,
>>
>
> Encoders do not emit RTP packets.  They emit encoded video frames.  Those
> are then packetized into RTP packets.  I'm suggesting the JS/wasm have
> access to the encoded frame before the packetization.
>
>
>
> Actually, encoders usually take raw framebuffers (4:2:0, 4:4:4 or other
> formats) + metadata and emit video frames + metadata.
>
It may be crucially important to get a handle on what the metadata looks
> like, in order to make sure we are able to transport not just the bytes of
> the frame, but the metadata too.
>


Yes, I agree.  My idea of an "encoded frame" includes metadata, such as
this:

https://cs.chromium.org/chromium/src/third_party/webrtc/common_video/include/video_frame.h?type=cs&q=EncodedImage&sq=package:chromium&l=28


> Metadata includes things like timing information (carried through the
> encoding process), interframe dependencies (an output from the encoding
> process) and preferences for encoding choices (input to the encoding
> process).
>
> We need to make sure we're not imagining things to be simpler than they
> actually are.
>

>
>
>> just wanted to check if it was a must-have requirement or a nice-to-have.
>> The full frame access is already used on mpeg-dash (IIRC) so that should be
>> supported, and I would also like to have an stream-pipeline like api in
>> which rtp packets don't need to go via the app to be forwarded from the
>> encoder to the rtp transport to the dtls transport to the ice transport..
>>
>> Best regards
>>
>> Sergio
>>
>>
>> On 05/03/2018 23:53, Peter Thatcher wrote:
>>
>> However the JS/wasm wants :).  It could use one frame == one QUIC stream
>> or one RTP packet == one QUIC stream or even one big RTP packet == one
>> frame == one QUIC stream.  I prefer the first, but the API should allow any
>> of these.
>>
>>
>> On Mon, Mar 5, 2018 at 2:50 PM Sergio Garcia Murillo <
>> sergio.garcia.murillo@gmail.com> wrote:
>>
>>> On 05/03/2018 23:22, Peter Thatcher wrote:
>>> > If you want to send media over QUIC or do your own crypto between
>>> > encode and network (perhaps over some low-level RTP transport), then
>>> > you need access to media after its encoded and before it's decoded.
>>>
>>> Peter, one side-question, how do you envision that media over quic
>>> should be used? Do you plan to encapsulate RTP over quic or just send
>>> each full frame over a single quic stream? Just trying to imagine what
>>> api surface should the encoders/decoders expose.
>>>
>>> Best regards
>>>
>>> Sergio
>>>
>>>
>>>
>>
>