Some notes on Mozilla's testing framework

I have a long standing action to take a closer look at Mozilla's testing framework to get some practical experience. Here are some notes.


Testing methods
=====
A few methods are implementation specific and out of scope for us (either because they are really out of scope or because we cannot rely on them being available on an interoperable basis, as of today at least):
- xpcshell
- compiled-code tests
- chrome tests
- browser chrome tests
- crash tests

The tests that are right in scope for us are:
- Mochitest
  https://developer.mozilla.org/en/Mochitest
  * close to pure JavaScript tests, although they may use some elevated privileges (the "waitForFocus" method in SimpleTest or the EventUtils class to emulate user interaction for instance). These privileges are not mandatory though.
  * based on MochiKit, a JavaScript framework
  * server-side execution done in a JavaScript Web server using the Mozilla platform. There, we'd have an actual Web server.
  * tests run in an iframe, test runner collects reports from SimpleTest.js:
   http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js
  * SimpleTest provides a slightly easier syntax for JavaScript tests than testharness: "ok", "is" (and "todo" for upcoming tests) as main test methods. It's not function oriented  as testharness.
  * less strict equality rules. Using "==" instead of "===", so type coercion at stake. I guess I'm a bit old-school here as I find it strange to see the following pass:
   is("", 0, "Empty string must be 0");
  * hundreds of tests available (DOM core, canvas, media, localStorage, etc) although it's hard to tell what is thoroughly test and what is not
  * separation between regression tests (on specific bugs) and feature tests is possible as files are named accordingly.
  * no separation between tests that require elevated privilege and those that don't from what I can tell.

- reftest
  * automation requires a specific build (no "--disable-tests" compile option)
  * alternatively, Mochitests using elevated privileges could be used (through snapshotWindow and compareSnapshots helper functions)


Desktop / Mobile
=====
- Same test tools ported to mobile version (page seems outdated though):
  https://wiki.mozilla.org/Mobile/Fennec_Mochitest
- Note memory problems to run all test at once
- I managed to compile a desktop version of Firefox Mobile. I did not manage to compile a working version for the Android emulator, and I do not have an Android >=2.0 device at hand. It might be normal because the emulator uses ARMv6 and that's not supported anymore (I tried the "--disable-thumb2" option without much success).
- In short, I did not succeed running tests there.


A few useful functionalities
=====
- For reftests, the possiblity to pass based on equality or on difference.
- For reftests, invalidation tests to check that repaints work as expected when something get changes through scripts after initial rendering. An additional event (MozReftestInvalidate in the case of Firefox) is needed to say "initial rendering done", coupled with a class="reftest-wait" attribute on the html element so that the test harness knows that it needs to wait. I don't know of any way to do that in a inter-operable way though.


Resources
=====
- Mozilla automated testing:
  https://developer.mozilla.org/en/Mozilla_automated_testing

- Overview of existing automated tests and frameworks on WHATWG wiki:
  http://wiki.whatwg.org/wiki/Browser_Tests_Project#Overview_of_existing_automated_tests_and_frameworks

- Mochitests SpecialPowers object for examples of elevated privileges:
  https://developer.mozilla.org/en/SpecialPowers

Received on Thursday, 21 April 2011 12:52:02 UTC