Re: Requiring tests for spec changes

I'm in favour of this. It'll also help us in the case where the meaning of
the spec text isn't clear --- at least there's an alternative description
of the expected behaviour.

Simon

On Mon, Sep 4, 2017 at 11:31 AM, James Graham <james@hoppipolla.co.uk>
wrote:

> Now that L1 is mostly done, it's time to start thinking about how we can
> learn from the experience and improve the process for future work.
>
> One thing that is being adopted by other specs to great success is to
> require that spec changes come with an accompanying test change, where
> possible (i.e. excluding things that don't affect the normative
> requirements). This has a number of advantages:
>
> * We have confidence that the spec has at least basic tests for everything.
> * We don't get stuck in the current situation of trying to retro-fit tests
> onto an existing spec.
> * It is immediately clear to implementors if a spec change requires a
> corresponding change in their implementation.
> * It is clear to spec authors if they are introducing a change that
> unexpectedly breaks existing implementations.
>
> I think the last points in particular are compelling. Although this seems
> like more work upfront, the feedback I have heard from specs like HTML and
> DOM is that requiring tests along with spec changes speeds up the overall
> specification process, because it's more likely that changes stick and
> aren't reverted later due to unforeseen compat issues. It also means that
> specification changes are more likely to reflected in implementations
> quickly (since there's an obvious failing test to fix), so it helps avoid
> the situation where a change is made and then implementations do nothing
> (because they simply didn't notice) and eventually it is reverted (or,
> worse, some implementations notice and make a fix and others don't, and
> clients end up relying on the behavioural differences).
>
> I strongly recommend that we adopt this requirement or writing tests for
> all post-L1 spec changes.
>
>