W3C home > Mailing lists > Public > public-test-infra@w3.org > October to December 2020

Automated Test Coverage for IDL specs

From: Dominique Hazael-Massieux <dom@w3.org>
Date: Thu, 10 Dec 2020 14:22:26 +0100
To: public-test-infra@w3.org
Message-ID: <ca530a29-dadc-7da8-4dd5-9b079db6519e@w3.org>

Following the conversations at TPAC around test plans and coverage [1],
I have been prototyping a tool to measure which parts of a spec IDL
surface get exercised in its test suite:

It uses JS proxies to wrap IDL-identifiable items and records when each
of them is exercised; right now, it is using Puppeteer to load pages
with that wrapper.

I have attached the JSON structure that describes which files in
wpt/webrtc hit which items of the WebRTC spec IDL (and how many times
they do).

The tool is incomplete (it only deals with the less exotic part of
WebIDL) and has bugs, as you would expect from a prototype. But the
approach seems pretty promising to help identify gaps or weaknesses in
test coverage.

I'll likely continue developing this for a bit to help me keep track of
the coverage of WebRTC specs. But I'd also be happy to discuss making
this a more structural part of WPT if there is any interest (which
presumably would involve moving away from puppeteer to wptrunner).


1. https://www.w3.org/2020/10/TPAC/breakout-schedule.html#wptcoverage

Received on Thursday, 10 December 2020 13:22:31 UTC

This archive was generated by hypermail 2.4.0 : Thursday, 10 December 2020 13:22:32 UTC