Re: Raw data API - 3 - Encoded data

On 29/05/2018 13:09, Harald Alvestrand wrote:
> Den 29. mai 2018 12:21, skrev Sergio Garcia Murillo:
>> In order to be able to implement this API on the browser, the codec
>> packetization of the encoded stream must be known by the browser.
> Why? This particular API proposal was intended to avoid having that
> property.
>
> I agree that fitting it on the RTPSender doesn't seem like a Good Idea;
> it may fit onto a GenericSender that is a superclass of RTPSender, or it
> may be designed as a "mixin" API that we can apply to both RTPSender and
> QUICSender (if such a beast exists), or "something else".

That's what I was against in my original message, RTP has some intrinsic 
that we can't ignore. Sending a binary frames over QUIC or datachannels 
or websockers is perfectly fine and do not require any special apis.

>> Also,
>> it is not possible to modify the frame, for example applying end to end
>> media encryption, as packetization requires to have access to the raw
>> data or add metadata for the very same reason.
> Does it?
> The H.264 packetization mode 0 and 1 packetizers (those are the ones
> I've done code for recently) need to ensure that they know about frame
> boundaries, and the frame needs to be encoded either with or without the
> 00 00 01 resync sequences, and there's numbering that has to be applied
> correctly, but otherwise, I'm not aware that the packetizer needs to
> know about the frame content.

If you are going to packetize after encryption, the frame must be 
already in annex b format as you can't search for the 0x00001 sequences. 
Also, you will have to pass the nal unit type, start position and length 
of each NAL unit to be able to decide which nal type to use in the rtp 
packetization (keep it, STAP-A or FU-A).

Even when passing the nal structure, the h264 packetization will not 
send the encoded byte for the nal unit header for non STAP-A packets 
(either "normal" nat unit, or FU unit), but will just send the nal type. 
So when reconstructing the payload, the depaketizer will insert a 
reconstructed and unencrypted NAL unit header in the middle of the of 
the encrypted frame, which will prevent it from working.


>
>> If the API would only alow to forward the frames from an encoder/decoder
>> (or another media source/sink) to the rtp objects, I would prefer a
>> higher level API that deals with streams and not individual packets.
> We've been asked to provide APIs that allow us to deal with the
> individual bytes. This is one option.
> How would you like an interface to look like?

If you need to deal with the individual bytes, in my opinion we should 
use the raw rtp packet api instead, you would just need to add the rtp 
packetization code in js (which could be custom btw)

Best regards
Sergio

Received on Tuesday, 29 May 2018 11:45:53 UTC