Re: Raw data API - 6 - Encoders/decoders

On Thu, Jun 14, 2018 at 1:55 AM Lorenzo Miniero <lorenzo@meetecho.com>
wrote:

> On Thu, 14 Jun 2018 10:18:34 +0200
> Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com> wrote:
>
> > On 14/06/2018 4:35, Peter Thatcher wrote:
> > > Yes, that's a valid concern.  But that concern is the same as for
> > > any time the app gets in the media flow, such as when it does
> > > processing for funny hats or adds encryption.  It might take some
> > > implementation experience to figure out where the really perf
> > > issues are. [...]
> > >  If the app can't process things per-frame, then you might as well
> > > just go back to having an RtpSender like in ORTC that doesn't
> > > require the app to do anything on the media path.
> > >  [...]
> > > But that requires every piece to be implemented by the browser, and
> > > I don't see how that buys us much vs. an ORTC-style RtpSender.   It
> > > would be much better if we could find a way to make wasm/js
> > > performant in the media path.
> > > [...]
> > > It would be easy to make the encoders and decoders use WHATWG
> > > streams instead of events.  I just don't see the benefit of having
> > > an encoder stream tied to a transport stream with no app in between
> > > except plugging it together and then hoping that it will be
> > > performant because we expect a sufficiently smart browser
> > > implementation.
> >
> > I think we have a mayor disagreement here. Your assumption is that
> > webrtc wants the low level APIs because they want to do funny things
> > with the raw data. IMHO that is not true.
> >
> > WebRTC developers just hate SDP, dislike transceivers, senders and
> > receivers and trying to get current APIs to work is a pain even for
> > the simple cases. ORTC is just a bit better, but not much, with the
> > encoding parameters stuff being just a  json-version of an m-line,
> > not much improvement there.
> >
> > So, after failing to provide a high level API that is usable,
> > developers are requesting a lower level API with a sane API, not
> > because they all want to deal with raw rtp packets, do funny hats or
> > send rtcp, but because we want to use something that is simpler, more
> > fine grained and with a better control surface. Some of them may want
> > to also perform raw operations in one way or another, so we should
> > also support that use cases.
> >
>
>
> +1 on all of the above...
>
>
> > In that regards the using whatwg streams provides a simple API
> > surface with finer grained control of each of the components in the
> > pipe line. The same stream API will most probably used in other areas
> > of the browser as well, so the web developers will already have
> > knowledge about it. Also, it allows to implement  the raw use cases
> > that we were trying to implement, even some of them (custom encoders,
> > funny hats) are explicitly mentioned in their documentation as a
> > target of the API.
> >
> > Does this API provide any advantage over the ORTC-sytle RTPSender if
> > we just don't want to do any raw processing? Yes, I would even
> > re-implement a standard rtp stack on js than having to use the ORTC
> > sender/receiver stuff.
> >
>
>
> -1 on an RTP stack in JS :-P


> I'm strongly against all this wasm nonsense. Just because machines are
> getting more powerful, doesn't mean we should drop more efficient code
> for something that runs worse just because it's easier to move around.
> It's a trend that is happening everywhere, and I hate it, because it
> shows no regard for those who still ship machines that would atthe very
> least struggle with that. Besides, this seems to bring us back to the
> "WebRTC just for browsers" era: WebRTC is everywhere, now, and I can
> already picture my mobile phone melting down any time I open a web page
> with WebRTC on it.
>

This is mainly about how low-level we want an RTP component to be.  There's
a balance between giving the app control/flexibility and ease of use for
the web developer, and there are perf considerations as well.

But allowing the web app to control RTP packetization is hardly an entire
RTP stack in JS.  It's just one small part of the whole.


> Even if all that weren't an issue, I still have strong concerns about
> the impact this would have on interoperability. Right now, no matter
> what you're using, as long as you implement the right protocols things
> can exchange media with each other. As Sergio said in another response,
> the ability to do your own transport protocol over whatever, or even
> your own RTP and packetization for that matter, will very likely lead to
> isolated silos that will only talk to what you implement. I'm sure
> that's just fine for the likes of Google, Cisco, etc.; very less so for
> pretty much everyone else.


Interop is an application choice.  Some will choose to, some will choose
not to.  But it doesn't make sense to tell the apps that don't want to
interop that they can't do certain things because it wouldn't interop.
They don't want to interop in the first place.  And giving apps the ability
to do different things doesn't stop you from interoping.


As a WebRTC server developer, I'm most
> definitely not going to catch up with any new "better than anything
> else around" protocol that would come out of that.
>

f someone else finds that doing things in a different, non-interop way is
better for there use case, that doesn't affect you or your use case.


>
> If "funny hats" and the like are the target here, as explained by
> others I think there are better (and basically ready)


"Funny hats" is only use case out of many.  Saying this is just about funny
hats is really ignoring a lot of important use cases.


> ways to accomplish
> this without disrupting the work that has been done in the last years.


>
> > Last point, the whatwg group has been working on the API for years
> > now, so we can leverage on their work and cut down the time to market
> > of the webrtc nv API.
> >
> > Let's put it the other way, what are the benefits of your API
> > compared to the whatwg streams?
> >
> > Best regards
> > Sergio
> >
> >
>
> I think Peter answered that with the "requires every piece to be
> implemented by the browser" argument, which is not a good enough reason
> for me. I understand that maintaining a WebRTC stack in C/C++ is
> complicated, but it's worth it. WebRTC hid most of the lowest level
> stuff (e.g., sending and receiving UDP) within the browser for many
> reasons (like preventing DoS attacks originated by JavaScript), and all
> this wasm nonsense to do everything in JavaScript brings us several
> steps back, meaning that at the very least those same arguments should
> be dealt with once more before anything happens.
>

I do think it's interesting to allow JS/wasm to do things if they want to
(codecs, jitter buffers, FEC, etc), but nothing I'm suggesting *requires*
it from the app, it only allows it to do so if it chooses.

The one exception is that I've suggested we could choose an RtpTransport
that requires the app in JS/wasm is RTP packetization (and RTCP messages).
But even there, I'm suggesting we have different levels of API we can
consider.  We can choose between the browser doing it or the app doing it,
or maybe allow both.

This really isn't "everything in the browser" or "everything in wasm".
This is "break the API into components such that the app can customize the
pipeline if it chooses, but can keep the heavy lifting in the browser if it
wants to"



>
> Of course, just my two cents,
> Lorenzo
>
> --
> I'm getting older but, unlike whisky, I'm not getting any better
> https://twitter.com/elminiero
>