Re: Testing for WebRTC

Hi there,

I was pointed at this list posting by jgraham, and I thought I'd chime
in with some relevant info on what Mozilla is doing in the WebRTC
testing space. We haven't approached automated interoperability testing
directly, but we're doing slightly more complex testing now than we used to.

Our current state of automated WebRTC testing consists of two test
harnesses: Mochitest and Steeplechase. Mochitest may be familiar to some
of you, it's Mozilla's standard web content test harness. We have a
number of WebRTC Mochitests[1] which test the various exposed APIs by
instantiating two PeerConnections on the same page and connecting them
to each other. This is enough to test the DOM plumbing, but not much of
the network layer.

In order to test the networking layer more fully, and to test the rather
involved set of NAT traversal code, we spent some time recently building
a new test harness we're calling Steeplechase[2]. Steeplechase is
designed to run tests on two separate client machines and provides a
signalling channel (using socket.io) that the tests can use to
communicate information to connect a pair of PeerConnections (as well as
anything else they need to communicate out-of-band). We did some
refactoring to make it possible to run some of our existing WebRTC
Mochitests in the Steeplechase harness, with half of the test running on
one client and the other half running on the other client. We have some
testing machines configured in Amazon EC2 with a very specific network
configuration with each client behind a separate NAT and a controller
machine that executes the test harness and can send commands to the
clients to run a browser.

We've discussed extending Steeplechase to run tests between different
Firefox versions, to catch regressions that would break cross-version
connectivity. This would be fairly straightforward. Extending it to work
cross-browser would be possible but quite a bit harder, since we have
lots of experience automating Firefox but not much experience automating
other browsers. If that work happened I would expect it to involve us
working with other browser vendors.

If we weren't concerned with fully-automated testing it would be a lot
simpler to extend the Steeplechase harness to have a "semi-automated"
mode, where the harness simply started a web server and provided two
URLs to load in a pair of browsers. The HTML+JS bits of the harness are
fairly straightforward and would not be difficult to make fully
cross-browser. The only Firefox-specific bits in there right now are for
loading some test configuration (which could simply be URL parameters
instead) and quitting the browser (which isn't necessary for
semi-automated testing).

-Ted

1. http://mxr.mozilla.org/mozilla-central/source/dom/media/tests/mochitest/
2. https://github.com/luser/steeplechase/

Received on Wednesday, 23 October 2013 19:12:35 UTC