wptrunner (web-platform-tests runner for Mozilla)

I have just pushed wptrunner, a web-platform-tests test runner to PyPI. 
This is designed to allow fully automated running of the tests in 
various Mozilla products (theoretically desktop Firefox, Firefox OS 
devices and Servo, in decreasing likelyhood of actually working). 
However the architecture is such that it may be possible to use for 
other browsers with a relatively small amount of work.

The runner depends on Python 2.7 (and I assume below that you also have 
virtualenv). It has only been tested on Linux although there is no known 
reason it shouldn't work on OSX too (it does when run directly from the 
Mozilla tree, at least). Windows support is on the roadmap but since 
it's untested there it's unlikely to work. In general the software is 
beta quality and so might behave unexpectedly, up to and including 
creating a tiny black hole that eventually engulfs the earth.

In order to install and run the harness with desktop Firefox, do 
something like the following:

mkdir wptrunner

cd wptrunner/

virtualenv .

. bin/activate

pip install wptrunner

mkdir data

update.py config/config.ini data

runtests.py --binary=/home/jgraham/local/firefox/firefox data/metadata 
data/tests/

Note that the command line still has rough edges and is likely to change.

The update.py step is responsible for grabbing a copy of the test 
repository and generating the manifest file. runtests.py actually runs 
the tests. By default this uses 1 process, but passing --processes=N 
allows multiple browsers to be run simultaneously which will 
substantially decrease the running time (at the cost of greater 
instability). The default output format is something we call "structured 
logs". This is useful for post-processing but rather hopeless for 
humans; I suggest using --log-raw=output.log --log-mach=- to get more 
readable terminal output.

Architecturally wptrunner is based on Marionette which is the Gecko 
implementation of WebDriver (except when running against Servo, which 
doesn't have WebDriver support). The Marionette API doesn't *quite* 
match the WebDriver python client, so porting to other WebDriver 
implementations won't be *entirely* trivial, but shouldn't be too hard 
either. Basically each supported product needs to have a Browser 
subclass, which knows how to launch the product with the webdriver 
server listening on a specified port. Then the Executor subclass has to 
actually connect to that browser on that port and start feeding it 
tests. If there is serious interest in reusing this code for more than 
just Mozilla, we can discuss how to achieve this, and how to make 
different implementations more pluggable.

Received on Tuesday, 1 April 2014 15:44:41 UTC