Re: webrtc nv and whatwg streams

On 6/9/2018 4:23 AM, Sergio Garcia Murillo wrote:
> I have been reading the whatwg stream api
> https://streams.spec.whatwg.org and it seems it already covers quite a 
> lot of our use cases. Copy&paste follows:
>
> By providing the foundation for these streams to be exposed to 
> developers, the Streams Standard enables use cases like:
>
>   * Video effects: piping a readable video stream through a transform
>     stream that applies effects in real time.
>
> Some examples of potential transform streams includes
>
>   * A video decoder, to which encoded bytes are written and from which
>     uncompressed video frames are read
>
> wouldn't make sense to try to leverage their work?
>
> source=(stream)=>encoder=(stream)=>packetizer=(stream)=>rtpsender=(stream)=>dtls/ice 
> transport
> or
> source=(stream)=>encoder=(stream)=>quic

Not to belabor a point here, but if this is to occur on the main thread 
there's a real problem with this API (and this could occur I suppose, 
with a bunch of additional work elsewhere in the spec, on a Worker).

Even on a Worker, outside of emscriptem/WASM code that carefully doesn't 
throw garbage, there will be GC pauses causing hiccups in the stream - 
potentially long (in a realtime sense).

This especially applies to anything wanting to do transforms to the 
data.  We did propose some APIs for doing transforms on video/image data 
in efficient ways, though that has stalled.


-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email randell-ietf@jesup.org!  Way too much spam