[web-platform-tests] Pull Request: Add coverage report and tools for WebRTC tests

wpt-pr-bot has just labeled a pull request from soareschen for https://github.com/w3c/web-platform-tests as "wg-webrtc":

== Add coverage report and tools for WebRTC tests ==
This PR is an experimental attempt to better keep track of the test coverage status of web platform tests for WebRTC.

### Motivation

Keeping track of the coverage status of specs can be difficult. Specs and tests are often updated independently and can get out of sync quickly. When reviewing tests or specs, we'd like readers from both sides to quickly determine how a normative statement is linked to a test case. A test reviewer should be able to find out which statement in the spec a test case is testing on, without having to read through the entire spec. Similarly a spec reviewer should be able to find out whether a normative statement has been tested and which test file contain the test cases, without having to read through all test cases.

When the spec is updated, the corresponding tests may not be updated immediately to reflect the changes. Contributors should be able to find out which test need to be updated since the last update, so that they can work on updating the tests.

Adding coverage status to version control system can help us keep track of whether the test coverage has improved over time as new tests are added, or whether coverage get worse if the tests lag too far behind the spec.

At the same time, a "test as you commit" policy is being adopted in WebRTC (w3c/webrtc-pc#1617). Ideally we'd like a future where tests are added as soon as the spec is updated. But until then we still have to keep track of the existing statements and add new tests until we reach 100% coverage.

### Coverage Report

A `coverage` directory is added under `webrtc` containing YAML files that report the test status of each normative statement in WebRTC. The YAML format is chosen so that it is friendly to human readers, software tools, and version control systems.

### Coverage Tools

A `tools` directory is added with simple Node.js scripts to report the coverage status of the tests. Following is the overall coverage status as of now:

```bash
$ cd webrtc/tools
$ node scripts/overview.js 
Overall Coverage
====================
todo        |    248
tested      |    315
trivial     |    173
untestable  |     79
====================
total       |    815
coverage    | 69.57%
====================
```

### Prior Art

This PR does several improvement over the current coverage report done by @rwaldron at https://rwaldron.github.io/webrtc-pc/. Writing the report in YAML enable us to better parsing, editing, version control, and helps keep in sync with the spec updates. We also link tested statements to the particular test files that contain the tests. I also feel that decluttering the coverage report with non-essential parts of the spec improves readability for the purpose of test development.

### Discussion

The approach I tried here is not perfect. There may be many unresolved issues and thus is not suited for the broader projects in WPT. As for now, the major contribution for this is for the first time we are able to tell that the coverage for WebRTC is around 69.57%, and that there are around 248 normative statements that still need to be tested.

There may be disagreement on whether the status of a normative statement is classified correctly in the coverage report. We also haven't updated the references in the existing test files to the latest editor's draft. All these can be done in separate PR/issues if we decide to merge this.

Any suggestion or feedback is welcomed. If anything we can have discussions during the upcoming TPAC.

See https://github.com/w3c/web-platform-tests/pull/8051

Received on Friday, 3 November 2017 07:00:56 UTC