Re: webrtc nv and whatwg streams

> Le 9 juin 2018 à 10:41 AM, Randell Jesup <randell-ietf@jesup.org> a écrit :
> 
> On 6/9/2018 4:23 AM, Sergio Garcia Murillo wrote:
>> I have been reading the whatwg stream api
>> https://streams.spec.whatwg.org <https://streams.spec.whatwg.org/> and it seems it already covers quite a lot of our use cases. Copy&paste follows:
>> By providing the foundation for these streams to be exposed to developers, the Streams Standard enables use cases like:
>> 
>> Video effects: piping a readable video stream through a transform stream that applies effects in real time.
>> Some examples of potential transform streams includes
>> 
>> A video decoder, to which encoded bytes are written and from which uncompressed video frames are read
>> wouldn't make sense to try to leverage their work?
>> 
>> source=(stream)=>encoder=(stream)=>packetizer=(stream)=>rtpsender=(stream)=>dtls/ice transport
>> or
>> source=(stream)=>encoder=(stream)=>quic
> 
> Not to belabor a point here, but if this is to occur on the main thread there's a real problem with this API (and this could occur I suppose, with a bunch of additional work elsewhere in the spec, on a Worker).  
> 
> Even on a Worker, outside of emscriptem/WASM code that carefully doesn't throw garbage, there will be GC pauses causing hiccups in the stream - potentially long (in a realtime sense).

I agree this is an important point to study.
To go off the main thread, the idea is probably to pipe the stream from a source to a consumer.
Given the consumer will have locked access to the source, the browser could optimize the data transfer.
This might work if the consumer is implemented in native code, not if JS.
Might be doable for WASM.

> 
> This especially applies to anything wanting to do transforms to the data.  We did propose some APIs for doing transforms on video/image data in efficient ways, though that has stalled.

Do you have pointers?

> 
> 
> -- 
> Randell Jesup -- rjesup a t mozilla d o t com
> Please please please don't email randell-ietf@jesup.org <mailto:randell-ietf@jesup.org>!  Way too much spam