Comparing Blink code coverage provided by web tests & WPT

This research uses the Chromium code coverage toolchain
<https://chromium.googlesource.com/chromium/src/+/master/docs/code_coverage.md>
to gather and compare the code coverage of Blink provided by web tests &
WPT (using run_web_tests.py and content_shell). Three runs were done at the
same revision: the full web tests, WPT only (external/wpt), and web tests
without WPT. Virtual tests are included/excluded accordingly.

The full results can be viewed at:
https://storage.googleapis.com/blink-wpt-coverage/201812/index.html

Overall, WPT provides lower coverage than other web tests. And here’s a
manually curated list of Blink components whose WPT coverage is much (>30%)
lower than the full web tests (bold components have an even larger
difference than the rest):

Accessiblity
> BackgroundSync
> Contacts
> Editing
> Forms
> ImageCapture
> Location
> Media>Controls
> Media>Session
> Media>Recording
> PresentationAPI
> PushAPI
> SVG
> SecurityFeature>CredentialManagement
> Sensor>DeviceOrientation
> Speech
> Storage>FileSystem
> Storage>Quota
> Vibration
> WebGL
> WebGPU
> WebMIDI
> WebXR>VR


Some of the differences are to be expected (e.g. WebGL/GPU is not included
in WPT at the moment). We will use look closer at the list to identify key
areas where we can improve testing infrastructure and automation.
Meanwhile, suggestions or insights about the data are much appreciated!

(Link to details about test setup
<https://docs.google.com/document/d/1D4qPWWkGSV-CpPnhAVAMDW3hQQtaJTO6YDVOCkGOUAo/edit#heading=h.gl0ch3w76fw>
.)

Robert

Received on Tuesday, 18 December 2018 18:33:48 UTC