Some testing results

I was planning to share this much earlier, but illness got the better of me last week and it slipped my mind.

Since the Chrome M23 announcement, I have spent a little time with Chromium.  My initial goal was to develop the same ICE lite + G.711 demonstration I shared earlier.  After all, the browser-to-browser cases have already been done to death.

My initial testing used Chromium 24.0.1284.0 (159474).  I've re-validated several of these issues with later versions, but not all.  

Firefox is unfortunately badly broken for me right now.  That's a shame because what I do see is promising, even though a lot of things are clearly not yet implemented (too much to make any real sort of judgment about, unfortunately).

Early indications are that the two implementations have no chance of interoperating at this point in time.  See below.

Here is what I have found.  I assume that a lot of these are known bugs.  There are some points where we need to reach some consensus on what the right behavior is.

1.	Chromium doesn't support ICE lite.  I found this odd, considering that this was after support for RFC 5245 was announced.
	a.	Any "a=ice-lite" line is stripped (silently) from answers.
	b.	Chromium refuses to play media unless it has received a valid check from the source of that media. !
	c.	(Unconfirmed) A comment to the mailing list indicates that Chromium can't act in the controlling role unless it generates the offer.  This would be consistent with not supporting ICE lite. 
	d.	I haven't confirmed if this also means that Chromium doesn't support ICE role conflict resolution.

2.	Chromium requires the "a=ssrc:... mslabel:..." line to play back media.  Without this option, no media plays.
	a.	This is very different to what is described in draft-alvestrand-mmusic-msid.
	b.	The cname and label ssrc parameters are provided also, but these are less critical.  Omitting these does not prevent playback.
	c.	The cname parameter doesn't appear to be necessary at all, it can even be empty with no observable consequence.
	d.	Chromium will create a media stream for every mslabel that is present, even if no media arrives for that tag.
	e.	The label parameter is used to label the track that is created.
	f.	Firefox does not generate these attributes.  That means Chromium users won't be able to see or hear Firefox users.

3.	Chromium generates an offer with a proprietary ICE option by default: "a=ice-options:google-ice".
	a.	I was able to remove this line successfully, just as I was able to generate an answer without the attribute.
	b.	This is listed as session level in the registry, but it appears at the media level.

4.	Initial offers generated by Chromium don't include transport parameters.  The c= and a=rtcp: lines include an invalid address (either 127.0.0.1 or 0.0.0.0).  The m= lines include a valid (but presumably unusable) port value of 1. 
	a.	These values are populated once candidates are gathered.  This is only accessible through "pc.localDescription.sdp".
	b.	This is what blah-mmusic-ice-trickle says, sure.
	c.	Even gathering host candidates takes a noticeable amount of time.  Around 500ms for UDP and another 500ms for TCP.  Having implemented this, I really don't understand why.
	d.	Completion of candidate gathering seems to be indicated with an "icecandidate" event containing a null candidate.  There is supposed to be some explicit attribute for this, but it's not there (or at least not enumerable).
	e.	All candidates have a non-specified "generation" attribute.  It's obvious that this is XEP-0176 related, but it's unclear why it is necessary to include this label.

5.	Chromium generates a lot of connectivity checks.  A lot more than would seem to be necessary for continuing consent.  One every 500ms in fact.  I'd be interested in learning why this was considered necessary.

6.	Chromium restarts ICE quite aggressively.  ~15 seconds of no STUN responses and it creates a new set of candidates and starts over.
	a.	Old candidates seem to be closed, but are still present in the offer/answer.  Is there a reason for this?  I can imagine that a peer that doesn't understand "generation" is going to get really confused (or slow) as a result of this.
	b.	New candidates get tagged with a monotonically increasing "generation".

7.	(Windows) Chromium by default picks up the default audio device as opposed to the default *communications* device for both input and output.  This is an interesting choice and I presume that it is based on the fact that the <audio> tag has to play from arbitrary sources and it can't be switched to a different output based on some esoteric platform-based configuration and better that the selection be consistent than have to deal with that.  I may be an unusual type of user in the sense that I have multiple sets of audio devices.
	a.	I notice that the Firefox nightly does not allow a choice at all, but plays back through the default audio device, while sourcing from the default communications device.  My configuration has that using two different headsets.

8.	Chromium always reports the iceState as "closed".  Even for a completely new RTCPeerConnection.  Is this simply not implemented yet?

9.	Chromium gives an error that seems to be related to URL.createObjectURL().  This does not have any effect on the session, but I get a 404 error on something like: blob:http%3A//localhost%3A9799/00c5178d-86eb-48b4-a92f-f2d3fa246b

10.	Closing an RTCPeerConnection twice triggers an exception.  Idempotence is a useful property in such a function.  Second time should do nothing.

11.	Chromium seems to accept any sort of gibberish in both offers and answers.  It then applies what it got prior to the error.  For example, I once mistyped an "a=ice-lite" line with a colon instead of '='.  This resulted in a session with no m= lines.  I did have an error callback set to console.log.bind(console), but I got neither error callback nor exception.  It would be good if we could (as Anant has proposed) get this part right.  Getting some certainty about what SDP is legal should help guide whether we should be generating errors or whether ignoring certain things is possible (with potentially some form of warning as appropriate).
	a.	Chromium doesn't support the p= or e= lines.  This also triggers the m= line disappearance problem.  On both local and remote. 

12.	Even just instantiating mumbleRTCPeerConnection was not obvious.  Some builds refused to take every combination of inputs that we tried: no args, null, empty dictionary, dictionary with empty iceServers.  All generated obtuse errors in some builds.
	a.	A colleague had massive problems with this, he continually gets mysterious error codes (mostly 9 and 17, depending on what arguments he passes the constructor) and is unable to instantiate it.  This worked fine on my machine, so it may be a configuration issue.

13.	Firefox Nightly will reject an m= line if it doesn't have media for that line.  Rather than accepting the media and "signaling" a=recvonly, it just refuses it.

14.	Firefox Nightly will refuse to play video in the absence of audio.  It ended up being a frustrating exercise working that one out.

All told, I found the experience of using this API to be far from straightforward.  Going into this, I expected that this would be pretty easy considering how well I *thought* that I understood the API.  

That is even making an allowance for the fact that there would be bugs - I fully appreciate the challenges of getting something of this complexity and magnitude working.  The ICE lite and mslabel issues in particular were hard to nail down until I went back to first principles.

I expected some trial and error, but I was surprised by how bad it ended up being.

--Martin

Received on Monday, 29 October 2012 05:17:50 UTC