[w3c/ServiceWorker] Safety versus simplicity in specification tests (#1087)

I'm working on [the Service Worker tests in the Web Platform Tests Project](https://github.com/w3c/web-platform-tests/tree/master/service-workers), and I was hoping to get some advice from specification authors and implementers.

While upstreaming tests from the Chromium project, I ran across a Chromium bug report regarding the potential for instability in a testing pattern:

https://bugs.chromium.org/p/chromium/issues/detail?id=558244

In short: because an "idle" service worker may be terminated at any time, tests that track state between events via global variables are susceptible to intermittent failure.

It turns out that there are already tests in Web Platform Tests that suffer from this problem. This has led me to research mitigation techniques. I first used `IndexedDB` to store state, but I'm reluctant to introduce dependencies on unrelated technologies. More recently, I've been experimenting with a simple "keep alive" protocol where the client uses `postMessage` on a regular interval, and the worker uses `waitUntil` on the message events to prevent termination. I described this in more detail while asking for feedback on the public-webapps mailing list:

https://lists.w3.org/Archives/Public/public-webapps/2017JanMar/0035.html

@asutherland confirmed that the approach was sound, but he questioned whether it was necessary, even suggesting that implementing it broadly would degrade the value of the tests:

https://lists.w3.org/Archives/Public/public-webapps/2017JanMar/0040.html

Web reality may also have bearing here. At face value, the specification makes no guarantees about when worker termination may occur. But Andrew tells me that in practice, browsers are not expected to attempt to terminate often enough to impact short-lived tests. And even if this were expected to be a problem, it may be reasonable for the Web Platform Tests project to request that consumers take special precautions to avoid termination while executing the tests.

So I'm left wondering: what is the right answer for the Web Platform Tests project? On the one hand, requiring tests to initiate this "keep alive" protocol will promote test stability. On the other hand, it increases test complexity in a way that may make the them harder to consume, all to guard against an impractical circumstance.

I think the answer requires an understanding of the specification, of implementation reality, and of policy/philosophy for the Web Platform Tests project. That's why I'm asking here and also pinging @Ms2ger, @jgraham, @zcorpan, and @annekv.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/w3c/ServiceWorker/issues/1087