Re: active speaker information in mixed streams

I don't think it needs to be an event. Just poll it at the frequency you
care about.


On Tue, Jan 28, 2014 at 5:26 PM, Peter Thatcher <pthatcher@google.com>wrote:

> On Tue, Jan 28, 2014 at 5:21 PM, Roman Shpount <rshpount@turbobridge.com>
> wrote:
> > Would it make more sense to generalize a RtpContributingSource to define
> a
> > list of RTP header extensions and trigger an event every time the value
> set
> > changes:
> >
> > dictionary RtpHeaderExtension {
> >   unsigned short id;
> >   ArrayBuffer value;
> > }
> >
> > dictionary RtpContributingSource {
> >   unsigned int csrc;
> >   sequence<RtpHeaderExtension> headerExtensions;
> > }
> >
> > This way it is not limited to audio level only.
> >
>
> Like Justin said, it's getting quite low-level at that point.  It's
> not much different than my "give JS access to every packet" event.
>
> > This being said, the only problem I see with all of this is that there
> are
> > scenarios (like audio level) when this event will be triggered for every
> > packet. This will not scale for server side applications of orca.
> >
>
> Since we only care about the latest values, can't we just throttle how
> often the event is fired?  Say, every 200ms?
>
> > _____________
> > Roman Shpount
> >
> >
> > On Tue, Jan 28, 2014 at 7:56 PM, Peter Thatcher <pthatcher@google.com>
> > wrote:
> >>
> >> Yes, it's pretty low-level.  For this particular use case, what you
> >> have is better, although I'm not sure I'd like calling it "MixerInfo".
> >>  How about just calling them "contributing source"s?
> >>
> >> dictionary RtpContributingSource {
> >>   unsigned int csrc;
> >>   int audioLevel;
> >> }
> >>
> >> partial interface RtpReceiver {
> >>   sequence<RtpContributingSource> getContributingSources();
> >> }
> >>
> >>
> >> Also, is it enough to require JS to poll?  Why not have an event for
> >> when the values change?
> >>
> >> partial interface RtpReceiver {
> >>    // Gets sequence<RtpContributingSource>
> >>    attribute EventHandler? oncontributingsources;
> >> }
> >>
> >>
> >> Even so, would it still be worth it to have low-level header extension
> >> access?  It might be handy when an application wants a proprietary
> >> header extension sent from their "mixer".  On the other hand, one
> >> could probably just use the data channel, like I suggested earlier :).
> >>
> >> By the way, the ease at which you put this on the RtpReceiver does
> >> show what an advantage it is to have it.
> >>
> >>
> >> On Tue, Jan 28, 2014 at 4:21 PM, Justin Uberti <juberti@google.com>
> wrote:
> >> > Having to mine through the raw packets feels like a pretty low-level
> API
> >> > to
> >> > me.
> >> >
> >> > I was thinking that one could interrogate the RtpReceiver object to
> get
> >> > data
> >> > on the most recently seen CSRCs and their corresponding energy levels.
> >> > Something like
> >> >
> >> > dictionary RtpCsrcInfo {
> >> >   unsigned int csrc;
> >> >   int audioLevel;
> >> > }
> >> >
> >> > dictionary RtpMixerInfo {
> >> >   sequence<RtpCsrcInfo> csrcs;
> >> > }
> >> >
> >> > partial interface RtpReceiver {
> >> >   RtpMixerInfo getMixerInfo();
> >> > }
> >> >
> >> > or maybe just return a dictionary with CSRC as keys and energy levels
> as
> >> > values.
> >> >
> >> >
> >> > On Tue, Jan 28, 2014 at 3:27 PM, Peter Thatcher <pthatcher@google.com
> >
> >> > wrote:
> >> >>
> >> >> I think it would be reasonable to add some access to header
> extensions
> >> >> and CSRCs in the RtpReceiver object.
> >> >>
> >> >>
> >> >> Would it make sense to have a general access to such things by having
> >> >> general access to receive packets?  It could be used like so:
> >> >>
> >> >> var receiver = new RtpReceiver(...);
> >> >> receiver.onpackets = function(packets) {
> >> >>   for (var i = 0; i < packets.length; i++) {
> >> >>     var packet = packets[i];
> >> >>     // Here you have access to
> >> >>     // packet.csrcs
> >> >>     // packet.headerExtensions
> >> >>   }
> >> >> }
> >> >>
> >> >> And defined like so:
> >> >>
> >> >> partial interface RtpReceiver {
> >> >>   // Gives a sequence of RtpPacket
> >> >>   // Fired in "batches" of packets.
> >> >>   attribute EventHandler? onpackets;
> >> >> }
> >> >>
> >> >> dictionary RtpPacket {
> >> >>   sequence<unsigned int> csrcs;
> >> >>   sequence<RtpHeaderExtension> headerExtensions;
> >> >> }
> >> >>
> >> >> dictionary RtpHeaderExtension {
> >> >>   unsigned short id;
> >> >>   ArrayBuffer value;
> >> >> }
> >> >>
> >> >>
> >> >> That might leave a bit of work for you to build on top of, but it
> >> >> would solve the "can I access header extension" issue once and for
> >> >> all.
> >> >>
> >> >> Would this meet your needs?
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Jan 28, 2014 at 2:51 PM, Emil Ivov <emcho@jitsi.org> wrote:
> >> >> > On Tue, Jan 28, 2014 at 11:41 PM, Peter Thatcher
> >> >> > <pthatcher@google.com>
> >> >> > wrote:
> >> >> >> I guess it could continue in both.  The ORCA  CG might be quicker
> to
> >> >> >> integrate something into the API than the WebRTC WG.
> >> >> >>
> >> >> >> My question is the same: exactly what info do you want available
> in
> >> >> >> the JS?  The CSRCs?
> >> >> >
> >> >> > Same answer then: That would be CSRCs and/or audio level header
> >> >> > extensions as per RFC6465.
> >> >> >
> >> >> > Emil
> >> >> >
> >> >> > --
> >> >> > https://jitsi.org
> >> >> >
> >> >> >> On Tue, Jan 28, 2014 at 2:38 PM, Emil Ivov <emcho@jitsi.org>
> wrote:
> >> >> >>> I am not sure whether this discussion should only continue on one
> >> >> >>> of
> >> >> >>> the lists but until we figure that out I am going to answer here
> as
> >> >> >>> well
> >> >> >>>
> >> >> >>> Sync isn't really the issue here. It's mostly about the fact that
> >> >> >>> the
> >> >> >>> mixer is not a WebRTC entity. This means that it most likely
> >> >> >>> doesn't
> >> >> >>> even know what SCTP is, it doesn't necessarily have access to
> >> >> >>> signalling and above all, the mix is likely to also contain audio
> >> >> >>> from
> >> >> >>> non-webrtc endpoints. Using DataChannels in such situations would
> >> >> >>> likely turn out to be quite convoluted.
> >> >> >>>
> >> >> >>> Emil
> >> >> >>>
> >> >> >>> On Tue, Jan 28, 2014 at 10:18 PM, Peter Thatcher
> >> >> >>> <pthatcher@google.com> wrote:
> >> >> >>>> Over there, I suggested that you could simply send the audio
> >> >> >>>> levels
> >> >> >>>> over an unordered data channel.  If you're using one
> >> >> >>>> IceTransport/DtlsTransport pair for both your RTP and SCTP, it
> >> >> >>>> would
> >> >> >>>> probably stay very closely in sync.
> >> >> >>>>
> >> >> >>>> On Tue, Jan 28, 2014 at 5:44 AM, Emil Ivov <emcho@jitsi.org>
> >> >> >>>> wrote:
> >> >> >>>>> Hey all,
> >> >> >>>>>
> >> >> >>>>> I just posted this to the WebRTC list here:
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>>
> http://lists.w3.org/Archives/Public/public-webrtc/2014Jan/0256.html
> >> >> >>>>>
> >> >> >>>>> But I believe it's a question that is also very much worth
> >> >> >>>>> resolving
> >> >> >>>>> for ORTC, so I am also asking it here:
> >> >> >>>>>
> >> >> >>>>> One requirement that we often bump against is the possibility
> to
> >> >> >>>>> extract active speaker information from an incoming *mixed*
> audio
> >> >> >>>>> stream. Acquiring the CSRC list from RTP would be a good start.
> >> >> >>>>> Audio
> >> >> >>>>> levels as per RFC6465 would be even better.
> >> >> >>>>>
> >> >> >>>>> Thoughts?
> >> >> >>>>>
> >> >> >>>>> Emil
> >> >> >>>
> >> >> >>> --
> >> >> >>> https://jitsi.org
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Emil Ivov, Ph.D.                       67000 Strasbourg,
> >> >> > Project Lead                           France
> >> >> > Jitsi
> >> >> > emcho@jitsi.org                        PHONE: +33.1.77.62.43.30
> >> >> > https://jitsi.org                       FAX:   +33.1.77.62.47.31
> >> >>
> >> >
> >>
> >
>