Test runner now available in web-platform-tests

A simple in-browser test runner is now available in the 
web-platform-tests repository. This will automatically run 
testharness.js tests and provides UI for manually marking the results of 
reftests and manual tests. This runner is designed to be helpful when 
developing tests and implementations, allowing all tests, or a subset of 
tests, to be run with minimal labour, and to be helpful in comparing 
various implementations.

To facilitate the latter use case, the runner can produce JSON output 
from the test run, and there is a simple command line script for taking 
this output from one or more browsers and producing an implementation 
report (this script could be ported to run entirely in-browser).

I suggest that henceforth we stop compiling implementation reports by 
hand on wiki pages and instead ask implementors to provide the JSON 
output from their implementation. This could be from the runner in the 
repository or from implementor-specific test harnesses used in 
automation. In the longer term we should look to get a vendor-controlled 
URL at which the latest test result data for each implementation is 
published. Given this we will be able to automatically collate a test 
report from the latest available data.

In order to use the in-repository runner it is necessary to:

a) Set up the latest web-platform-tests checkout, following the 
instructions in README.md to install all the relevant submodules and set 
up the hosts.

b) Generate a test manifest using

python tools/scripts/manifest.py MANIFEST.json

from the web-platform-tests root (this step must be repeated whenever 
new tests are added).

c) Start the local server using

python serve.py

b) Navigate to

http://web-platform.test:8000/tools/runner/index.html

This is all documented in the README.md file in the web-platform-tests 
repository.

Received on Monday, 13 January 2014 11:01:09 UTC