Requiring tests for spec changes

Now that L1 is mostly done, it's time to start thinking about how we can 
learn from the experience and improve the process for future work.

One thing that is being adopted by other specs to great success is to 
require that spec changes come with an accompanying test change, where 
possible (i.e. excluding things that don't affect the normative 
requirements). This has a number of advantages:

* We have confidence that the spec has at least basic tests for everything.
* We don't get stuck in the current situation of trying to retro-fit 
tests onto an existing spec.
* It is immediately clear to implementors if a spec change requires a 
corresponding change in their implementation.
* It is clear to spec authors if they are introducing a change that 
unexpectedly breaks existing implementations.

I think the last points in particular are compelling. Although this 
seems like more work upfront, the feedback I have heard from specs like 
HTML and DOM is that requiring tests along with spec changes speeds up 
the overall specification process, because it's more likely that changes 
stick and aren't reverted later due to unforeseen compat issues. It also 
means that specification changes are more likely to reflected in 
implementations quickly (since there's an obvious failing test to fix), 
so it helps avoid the situation where a change is made and then 
implementations do nothing (because they simply didn't notice) and 
eventually it is reverted (or, worse, some implementations notice and 
make a fix and others don't, and clients end up relying on the 
behavioural differences).

I strongly recommend that we adopt this requirement or writing tests for 
all post-L1 spec changes.

Received on Monday, 4 September 2017 10:32:03 UTC