- From: James Graham <james@hoppipolla.co.uk>
- Date: Mon, 13 Jan 2014 11:00:44 +0000
- To: "public-test-infra@w3.org" <public-test-infra@w3.org>, public-webapps@w3.org, public-html@w3.org, "public-web-perf@w3.org" <public-web-perf@w3.org>
A simple in-browser test runner is now available in the web-platform-tests repository. This will automatically run testharness.js tests and provides UI for manually marking the results of reftests and manual tests. This runner is designed to be helpful when developing tests and implementations, allowing all tests, or a subset of tests, to be run with minimal labour, and to be helpful in comparing various implementations. To facilitate the latter use case, the runner can produce JSON output from the test run, and there is a simple command line script for taking this output from one or more browsers and producing an implementation report (this script could be ported to run entirely in-browser). I suggest that henceforth we stop compiling implementation reports by hand on wiki pages and instead ask implementors to provide the JSON output from their implementation. This could be from the runner in the repository or from implementor-specific test harnesses used in automation. In the longer term we should look to get a vendor-controlled URL at which the latest test result data for each implementation is published. Given this we will be able to automatically collate a test report from the latest available data. In order to use the in-repository runner it is necessary to: a) Set up the latest web-platform-tests checkout, following the instructions in README.md to install all the relevant submodules and set up the hosts. b) Generate a test manifest using python tools/scripts/manifest.py MANIFEST.json from the web-platform-tests root (this step must be repeated whenever new tests are added). c) Start the local server using python serve.py b) Navigate to http://web-platform.test:8000/tools/runner/index.html This is all documented in the README.md file in the web-platform-tests repository.
Received on Monday, 13 January 2014 11:01:12 UTC