Re: To Stream or not to Stream

I'm glad we're sounding more on the same page now :).  We all want
lower-level components that can get assembled together by the app.   And
streams vs. not-streams is the secondary to the question of what these
components are and how they get connected.

The crux of the problem seems to be that we have two seemingly conflicting
properties:

1.  Apps that want to be in the media path can be.
2.  Apps that don't want to be don't have to be.

What I have proposed so far doesn't solve #2 (yet).  And your hope is that
WHATWG streams could solve #2.



OK, so let's try and solve #2 for one case and then try and expand that to
other cases.  Let's start with the split between Encoder and RtpSender.
We've already demonstrated how the app could be between Encoder and
RtpSender by pulling frames out of the encoder and putting them into the
RtpSender.  But what if it doesn't want to be?  I think we have a few
options:

A.  new method RtpSender.setEncodeer(encoder)
B.  new method Encoder.addSink(RtpSender)
C.  WHATWG streams: RtpSender.encodedFrames = Encoder.encodedFrames

I can see any of these happening.  But assuming they could all be
performant (and I'm still not sure C can be), it's probably mostly a matter
of taste/style.

Except for one thing: feedback.  For example, how do key frame requests and
bitrates get from the RtpSender to the encoder?  A seems like the best
approach for handling that.


OK, let's try another.  How about RtpReceiver and Decoder?

A.  new method RtpReceiver.setDecoder(decoder)
B.  new method Decoder.addSink(RtpReceiver)
C.  WHATWG streams: Decoder.encodedFrames = RtpReceiver.encodedFrames


Same problem: feedback.  For example, how does the RtpReceiver know when to
send a key frame requests?  And how does it get stats it might want to send
over RTCP?  A seems like the best approach for that.


What about splitting out the RtpTransport from the RtpSender?


A.  new method RtpSender.setTransport(transport)
B.  new method RtpTransport.addSender(RtpSender)
C.  WHATWG streams: RtpTransport.sendPackets = RtpSender.packets


Again, we have to worry about feedback from sink to source.  A seems the
best approach for that.



What about splitting out the RtpTransport from the RtpReceiver?  Not this
involves demuxing RTP as well.


A.  new method RtpReceiver.setTransport(transport, ... demux points ...)
B.  new method RtpTransport.addReceiver(RtpSender, ... demux points ...)
C.  WHATWG streams: new object RtpDemuxer(RtpTransport, demux points) plus
RtpReceiver.packets = RtpDemuxer.packets


Out of these, B actually seems the best to me in isolation, but I'd
probably want to be consistent with the other solutions (either A or B or
C).



OK, that was a fun brainstorming.  The conclusions I would come to at the
moment are:

1.  Even if we split out the RtpTransport from the RtpSender/RtpReceiver,
it seems to make sense to keep the RtpSender/Receiver around for those apps
which don't want to do packetization/demuxing on their own.

2.  We need to worry about not just the direction of media flow, but also
feedback flow (key frame requests, etc).

3.  A few simple methods to attach objects together might be sufficient to
allow apps that don't want to be in the media path not be.  WHATWG streams
might be as well if we figure out the feedback flow.













On Thu, Jun 14, 2018 at 2:09 PM Sergio Garcia Murillo <
sergio.garcia.murillo@gmail.com> wrote:

>
> >> IMHO the questions we have to answer are what components do we need, how
> >> do we link them together and how much low-level is enough, but, as good
> >> technicians we have jumped too soon into "show-me-the-code" mode ;)
> > That seems to me that you're singing from the same page as Peter, when
> > he says "wasting time on streams vs non-streams is useless", which is
> > exactly the opposite viewpoint that I read out of your earlier message.
> >
> > Thanks for clarifying - I was confused!
>
> I was also confused by seeing IDL definitions on the introductory
> emails, glad to know that we are on the same page and that we will leave
> the interface definitions for later.
>
> Best regards
> Sergio
>
>