Re: What would you like to see in WebRTC next? A low-level API?

Thank you for this fruitful discussion.

Yes, JavaScript programs cannot handle RTP-binary currently.

In the case which user handle data generated in JavaScript on Browser, 
the metadata plan would works perfectly.
In my case, we need to consider getUserMedia API or another API which 
offer tactile data to JavaScript programs.

I am sure you already know about this but video follows this flow.

(A-1)get raw data -> (A-2)encode -> (A-3)generate RTP packets -> 
(A-4)transmit(on MediaStream) -> (A-5)extract payload -> (A-6)decode -> 
(A-7)buffering -> (A-8)play

Also, tactile data follows this flow.

(B-1)get raw data -> (B-2)encode -> (B-3)packetize -> (B-4)transmit(on 
DataChannel) -> (B-5)de-packetize -> (B-6)decode -> (B-7)buffering -> 
(B-8)play

In the metadata plan, (B-1)processes in JavaScript programs need to set 
metadata to (A-1)process in Browser, and (B-7)processes also need to get 
metadata from (A-7)process.
Then browsers also need to offer methods and event triggers for 
(B-1)and(B-7).
Ideally, it is the best if Browsers have tactile encoders and decoders 
like video in the future.
Currently, my program gets tactile data through serial device API for 
Chrome App.

As for the getUserMedia, I have one more problem.
In my understanding, one getUserMedia API call grabs only one camera.
To make a 3D vision, videos from two eyes must be synchronized perfectly.
But the synchronization between medias from another getUserMedia API 
calls isn't supported.

Best Regards,
Toshiya Nakakura

On 2018年02月01日 20:41, Sergio Garcia Murillo wrote:
> AFIK the problem is that the rtp timestamp is not know until after the 
> video frame is encoded very deep on the video processing pipeline, so 
> it is not possible to retrieve it on js at sending time.
>
> Best regards
> Sergio
>
> On 01/02/2018 11:50, Lennart Grahl wrote:
>> I really am not an expert when it comes to A/V but there's a timestamp
>> field in the RTP... if the API would allow to retrieve that value
>> (.currentTimestamp) when sending and wait for it when receiving
>> (.waitForTimestamp(timestamp))... shouldn't that work?
>>
>> Cheers
>> Lennart
>>
>>
>> On 01.02.2018 08:22, Sergio Garcia Murillo wrote:
>>> If you can send a metadata payload within rtp, you can sync with
>>> external sources. Jus' sayin' ;)
>>>
>>> For example, if you create a json object with an uuid and send it via
>>> datachannel, you could send that uuid in the rtp metadata.
>>>
>>> Then on the receiving side you could get an  onmetadata event on the
>>> media stream track, retrieve the uuid sent by rtp and sync it with the
>>> data sent via datachannel.
>>>
>>> If the json is small, you could even send the full json on the rtp
>>> metadata and not even send it via datachannels.
>>>
>>> Best regards
>>> Sergio