Final Report

Hi all,

Over the last couple of months I've been working on improving the
WebVTT test suite at https://github.com/w3c/web-platform-tests.
This was funded by Apple in preparation for taking the WebVTT spec
towards a W3C standard.

This is my final report and while the test suite could do with further
tests for edge cases, all the core functionality of the spec is now
covered and the tests are organized into easy-to-assess categories.

I've completed the following tasks:
* Improved the directory layout of the test suite.
* Updated all existing tests to be spec-compliant.
* Added tests to cover all API functions.
* Improved test tools, to make tests more readable and thus maintainable:
  * `parsing/cue-text-parsing/buildtests.py` now supports python
escape sequences.
  * implemented `parsing/file-parsing/tools/build.py` to simplify
file-parsing tests.
* Implemented a test coverage tool in python:
  * it automatically checks every line of the file parsing part of the
specification for test coverage.
  * it allows us to prove 100% test coverage of the file parsing part
of the spec.
* Added rendering tests to cover all major rendering features.
* Implemented a test result categorization tool in order to give a
better overview of test results.

The following pull-requests are still outstanding, waiting for
approval by Simon:
* https://github.com/w3c/web-platform-tests/pull/5570
* https://github.com/w3c/web-platform-tests/pull/6336
* https://github.com/w3c/web-platform-tests/pull/6502
* https://github.com/w3c/web-platform-tests/pull/6503

The following is an example of test categorization for Firefox Nightly
56.0a1 (2017-07-09):
- parsing:  108/119 - 90.76%
  api:  56/89 - 62.92%
  rendering:    60/231 - 25.97%
- cues: 98/260 - 37.69%
- regions:  0/25 - 0.0%
- stylesheets:  42/166 - 25.3%
- VTTRegion:    0/8 - 0.0%
  VTTCue:   29/31 - 93.55%
- file-parsing: 39/48 - 81.25%
  cue-text-parsing: 69/71 - 97.18%
- region-settings-parsing:  0/6 - 0.0%
- cue-settings-parsing: 13/16 - 81.25%
- ::cue:    19/23 - 82.61%
- ::cue():  19/136 - 13.97%
- ::cue-region: 0/1 - 0.0%
- ::cue-region():   0/1 - 0.0%
- :past:    0/15 - 0.0%
- :future:  0/5 - 0.0%
- bidi: 3/8 - 37.5%
TOTAL:  224/439 - 51.03%

Thanks,
Benjamin Schaaf

Received on Thursday, 20 July 2017 14:10:05 UTC