Re: active speaker information in mixed streams

Polling is fine with me.  What about calling it RtpContriburingSource?
 Do you prefer that or MixerInfo?

On Tue, Jan 28, 2014 at 5:29 PM, Justin Uberti <juberti@google.com> wrote:
> I don't think it needs to be an event. Just poll it at the frequency you
> care about.
>
>
> On Tue, Jan 28, 2014 at 5:26 PM, Peter Thatcher <pthatcher@google.com>
> wrote:
>>
>> On Tue, Jan 28, 2014 at 5:21 PM, Roman Shpount <rshpount@turbobridge.com>
>> wrote:
>> > Would it make more sense to generalize a RtpContributingSource to define
>> > a
>> > list of RTP header extensions and trigger an event every time the value
>> > set
>> > changes:
>> >
>> > dictionary RtpHeaderExtension {
>> >   unsigned short id;
>> >   ArrayBuffer value;
>> > }
>> >
>> > dictionary RtpContributingSource {
>> >   unsigned int csrc;
>> >   sequence<RtpHeaderExtension> headerExtensions;
>> > }
>> >
>> > This way it is not limited to audio level only.
>> >
>>
>> Like Justin said, it's getting quite low-level at that point.  It's
>> not much different than my "give JS access to every packet" event.
>>
>> > This being said, the only problem I see with all of this is that there
>> > are
>> > scenarios (like audio level) when this event will be triggered for every
>> > packet. This will not scale for server side applications of orca.
>> >
>>
>> Since we only care about the latest values, can't we just throttle how
>> often the event is fired?  Say, every 200ms?
>>
>> > _____________
>> > Roman Shpount
>> >
>> >
>> > On Tue, Jan 28, 2014 at 7:56 PM, Peter Thatcher <pthatcher@google.com>
>> > wrote:
>> >>
>> >> Yes, it's pretty low-level.  For this particular use case, what you
>> >> have is better, although I'm not sure I'd like calling it "MixerInfo".
>> >>  How about just calling them "contributing source"s?
>> >>
>> >> dictionary RtpContributingSource {
>> >>   unsigned int csrc;
>> >>   int audioLevel;
>> >> }
>> >>
>> >> partial interface RtpReceiver {
>> >>   sequence<RtpContributingSource> getContributingSources();
>> >> }
>> >>
>> >>
>> >> Also, is it enough to require JS to poll?  Why not have an event for
>> >> when the values change?
>> >>
>> >> partial interface RtpReceiver {
>> >>    // Gets sequence<RtpContributingSource>
>> >>    attribute EventHandler? oncontributingsources;
>> >> }
>> >>
>> >>
>> >> Even so, would it still be worth it to have low-level header extension
>> >> access?  It might be handy when an application wants a proprietary
>> >> header extension sent from their "mixer".  On the other hand, one
>> >> could probably just use the data channel, like I suggested earlier :).
>> >>
>> >> By the way, the ease at which you put this on the RtpReceiver does
>> >> show what an advantage it is to have it.
>> >>
>> >>
>> >> On Tue, Jan 28, 2014 at 4:21 PM, Justin Uberti <juberti@google.com>
>> >> wrote:
>> >> > Having to mine through the raw packets feels like a pretty low-level
>> >> > API
>> >> > to
>> >> > me.
>> >> >
>> >> > I was thinking that one could interrogate the RtpReceiver object to
>> >> > get
>> >> > data
>> >> > on the most recently seen CSRCs and their corresponding energy
>> >> > levels.
>> >> > Something like
>> >> >
>> >> > dictionary RtpCsrcInfo {
>> >> >   unsigned int csrc;
>> >> >   int audioLevel;
>> >> > }
>> >> >
>> >> > dictionary RtpMixerInfo {
>> >> >   sequence<RtpCsrcInfo> csrcs;
>> >> > }
>> >> >
>> >> > partial interface RtpReceiver {
>> >> >   RtpMixerInfo getMixerInfo();
>> >> > }
>> >> >
>> >> > or maybe just return a dictionary with CSRC as keys and energy levels
>> >> > as
>> >> > values.
>> >> >
>> >> >
>> >> > On Tue, Jan 28, 2014 at 3:27 PM, Peter Thatcher
>> >> > <pthatcher@google.com>
>> >> > wrote:
>> >> >>
>> >> >> I think it would be reasonable to add some access to header
>> >> >> extensions
>> >> >> and CSRCs in the RtpReceiver object.
>> >> >>
>> >> >>
>> >> >> Would it make sense to have a general access to such things by
>> >> >> having
>> >> >> general access to receive packets?  It could be used like so:
>> >> >>
>> >> >> var receiver = new RtpReceiver(...);
>> >> >> receiver.onpackets = function(packets) {
>> >> >>   for (var i = 0; i < packets.length; i++) {
>> >> >>     var packet = packets[i];
>> >> >>     // Here you have access to
>> >> >>     // packet.csrcs
>> >> >>     // packet.headerExtensions
>> >> >>   }
>> >> >> }
>> >> >>
>> >> >> And defined like so:
>> >> >>
>> >> >> partial interface RtpReceiver {
>> >> >>   // Gives a sequence of RtpPacket
>> >> >>   // Fired in "batches" of packets.
>> >> >>   attribute EventHandler? onpackets;
>> >> >> }
>> >> >>
>> >> >> dictionary RtpPacket {
>> >> >>   sequence<unsigned int> csrcs;
>> >> >>   sequence<RtpHeaderExtension> headerExtensions;
>> >> >> }
>> >> >>
>> >> >> dictionary RtpHeaderExtension {
>> >> >>   unsigned short id;
>> >> >>   ArrayBuffer value;
>> >> >> }
>> >> >>
>> >> >>
>> >> >> That might leave a bit of work for you to build on top of, but it
>> >> >> would solve the "can I access header extension" issue once and for
>> >> >> all.
>> >> >>
>> >> >> Would this meet your needs?
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Tue, Jan 28, 2014 at 2:51 PM, Emil Ivov <emcho@jitsi.org> wrote:
>> >> >> > On Tue, Jan 28, 2014 at 11:41 PM, Peter Thatcher
>> >> >> > <pthatcher@google.com>
>> >> >> > wrote:
>> >> >> >> I guess it could continue in both.  The ORCA  CG might be quicker
>> >> >> >> to
>> >> >> >> integrate something into the API than the WebRTC WG.
>> >> >> >>
>> >> >> >> My question is the same: exactly what info do you want available
>> >> >> >> in
>> >> >> >> the JS?  The CSRCs?
>> >> >> >
>> >> >> > Same answer then: That would be CSRCs and/or audio level header
>> >> >> > extensions as per RFC6465.
>> >> >> >
>> >> >> > Emil
>> >> >> >
>> >> >> > --
>> >> >> > https://jitsi.org
>> >> >> >
>> >> >> >> On Tue, Jan 28, 2014 at 2:38 PM, Emil Ivov <emcho@jitsi.org>
>> >> >> >> wrote:
>> >> >> >>> I am not sure whether this discussion should only continue on
>> >> >> >>> one
>> >> >> >>> of
>> >> >> >>> the lists but until we figure that out I am going to answer here
>> >> >> >>> as
>> >> >> >>> well
>> >> >> >>>
>> >> >> >>> Sync isn't really the issue here. It's mostly about the fact
>> >> >> >>> that
>> >> >> >>> the
>> >> >> >>> mixer is not a WebRTC entity. This means that it most likely
>> >> >> >>> doesn't
>> >> >> >>> even know what SCTP is, it doesn't necessarily have access to
>> >> >> >>> signalling and above all, the mix is likely to also contain
>> >> >> >>> audio
>> >> >> >>> from
>> >> >> >>> non-webrtc endpoints. Using DataChannels in such situations
>> >> >> >>> would
>> >> >> >>> likely turn out to be quite convoluted.
>> >> >> >>>
>> >> >> >>> Emil
>> >> >> >>>
>> >> >> >>> On Tue, Jan 28, 2014 at 10:18 PM, Peter Thatcher
>> >> >> >>> <pthatcher@google.com> wrote:
>> >> >> >>>> Over there, I suggested that you could simply send the audio
>> >> >> >>>> levels
>> >> >> >>>> over an unordered data channel.  If you're using one
>> >> >> >>>> IceTransport/DtlsTransport pair for both your RTP and SCTP, it
>> >> >> >>>> would
>> >> >> >>>> probably stay very closely in sync.
>> >> >> >>>>
>> >> >> >>>> On Tue, Jan 28, 2014 at 5:44 AM, Emil Ivov <emcho@jitsi.org>
>> >> >> >>>> wrote:
>> >> >> >>>>> Hey all,
>> >> >> >>>>>
>> >> >> >>>>> I just posted this to the WebRTC list here:
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> http://lists.w3.org/Archives/Public/public-webrtc/2014Jan/0256.html
>> >> >> >>>>>
>> >> >> >>>>> But I believe it's a question that is also very much worth
>> >> >> >>>>> resolving
>> >> >> >>>>> for ORTC, so I am also asking it here:
>> >> >> >>>>>
>> >> >> >>>>> One requirement that we often bump against is the possibility
>> >> >> >>>>> to
>> >> >> >>>>> extract active speaker information from an incoming *mixed*
>> >> >> >>>>> audio
>> >> >> >>>>> stream. Acquiring the CSRC list from RTP would be a good
>> >> >> >>>>> start.
>> >> >> >>>>> Audio
>> >> >> >>>>> levels as per RFC6465 would be even better.
>> >> >> >>>>>
>> >> >> >>>>> Thoughts?
>> >> >> >>>>>
>> >> >> >>>>> Emil
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> https://jitsi.org
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Emil Ivov, Ph.D.                       67000 Strasbourg,
>> >> >> > Project Lead                           France
>> >> >> > Jitsi
>> >> >> > emcho@jitsi.org                        PHONE: +33.1.77.62.43.30
>> >> >> > https://jitsi.org                       FAX:   +33.1.77.62.47.31
>> >> >>
>> >> >
>> >>
>> >
>
>

Received on Wednesday, 29 January 2014 01:33:21 UTC