W3C home > Mailing lists > Public > public-test-infra@w3.org > October to December 2018

Comparing Blink code coverage provided by web tests & WPT

From: Robert Ma <robertma@chromium.org>
Date: Tue, 18 Dec 2018 13:33:11 -0500
Message-ID: <CAOPAaNJ_-1abHpEGDve_AUOiiAvtji38TzHkgrwWOiNZxzZBLw@mail.gmail.com>
To: blink-dev <blink-dev@chromium.org>, ecosystem-infra <ecosystem-infra@chromium.org>
This research uses the Chromium code coverage toolchain
to gather and compare the code coverage of Blink provided by web tests &
WPT (using run_web_tests.py and content_shell). Three runs were done at the
same revision: the full web tests, WPT only (external/wpt), and web tests
without WPT. Virtual tests are included/excluded accordingly.

The full results can be viewed at:

Overall, WPT provides lower coverage than other web tests. And here’s a
manually curated list of Blink components whose WPT coverage is much (>30%)
lower than the full web tests (bold components have an even larger
difference than the rest):

> BackgroundSync
> Contacts
> Editing
> Forms
> ImageCapture
> Location
> Media>Controls
> Media>Session
> Media>Recording
> PresentationAPI
> PushAPI
> SecurityFeature>CredentialManagement
> Sensor>DeviceOrientation
> Speech
> Storage>FileSystem
> Storage>Quota
> Vibration
> WebGL
> WebGPU
> WebXR>VR

Some of the differences are to be expected (e.g. WebGL/GPU is not included
in WPT at the moment). We will use look closer at the list to identify key
areas where we can improve testing infrastructure and automation.
Meanwhile, suggestions or insights about the data are much appreciated!

(Link to details about test setup

Received on Tuesday, 18 December 2018 18:33:48 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:14 UTC