Re: Final Report

Great progress on the test suite from where I stand!
We can do with a few more cases in some of the categories, but the
groundwork is laid.

Cheers,
Silvia.

On Fri, Jul 21, 2017 at 12:09 AM, Benjamin Schaaf <ben.schaaf@gmail.com> wrote:
> Hi all,
>
> Over the last couple of months I've been working on improving the
> WebVTT test suite at https://github.com/w3c/web-platform-tests.
> This was funded by Apple in preparation for taking the WebVTT spec
> towards a W3C standard.
>
> This is my final report and while the test suite could do with further
> tests for edge cases, all the core functionality of the spec is now
> covered and the tests are organized into easy-to-assess categories.
>
> I've completed the following tasks:
> * Improved the directory layout of the test suite.
> * Updated all existing tests to be spec-compliant.
> * Added tests to cover all API functions.
> * Improved test tools, to make tests more readable and thus maintainable:
>   * `parsing/cue-text-parsing/buildtests.py` now supports python
> escape sequences.
>   * implemented `parsing/file-parsing/tools/build.py` to simplify
> file-parsing tests.
> * Implemented a test coverage tool in python:
>   * it automatically checks every line of the file parsing part of the
> specification for test coverage.
>   * it allows us to prove 100% test coverage of the file parsing part
> of the spec.
> * Added rendering tests to cover all major rendering features.
> * Implemented a test result categorization tool in order to give a
> better overview of test results.
>
> The following pull-requests are still outstanding, waiting for
> approval by Simon:
> * https://github.com/w3c/web-platform-tests/pull/5570
> * https://github.com/w3c/web-platform-tests/pull/6336
> * https://github.com/w3c/web-platform-tests/pull/6502
> * https://github.com/w3c/web-platform-tests/pull/6503
>
> The following is an example of test categorization for Firefox Nightly
> 56.0a1 (2017-07-09):
> - parsing:  108/119 - 90.76%
>   api:  56/89 - 62.92%
>   rendering:    60/231 - 25.97%
> - cues: 98/260 - 37.69%
> - regions:  0/25 - 0.0%
> - stylesheets:  42/166 - 25.3%
> - VTTRegion:    0/8 - 0.0%
>   VTTCue:   29/31 - 93.55%
> - file-parsing: 39/48 - 81.25%
>   cue-text-parsing: 69/71 - 97.18%
> - region-settings-parsing:  0/6 - 0.0%
> - cue-settings-parsing: 13/16 - 81.25%
> - ::cue:    19/23 - 82.61%
> - ::cue():  19/136 - 13.97%
> - ::cue-region: 0/1 - 0.0%
> - ::cue-region():   0/1 - 0.0%
> - :past:    0/15 - 0.0%
> - :future:  0/5 - 0.0%
> - bidi: 3/8 - 37.5%
> TOTAL:  224/439 - 51.03%
>
> Thanks,
> Benjamin Schaaf
>

Received on Friday, 28 July 2017 01:22:48 UTC