Re: Requiring tests for spec changes

+1

I was actually going to bring this up once we got to the end of September.
Thank you for bringing this up.

Moving to this approach will also improve our ability to know when
implementations are out of sync with browsers when they import WPT.

David

On 4 September 2017 at 11:31, James Graham <james@hoppipolla.co.uk> wrote:

> Now that L1 is mostly done, it's time to start thinking about how we can
> learn from the experience and improve the process for future work.
>
> One thing that is being adopted by other specs to great success is to
> require that spec changes come with an accompanying test change, where
> possible (i.e. excluding things that don't affect the normative
> requirements). This has a number of advantages:
>
> * We have confidence that the spec has at least basic tests for everything.
> * We don't get stuck in the current situation of trying to retro-fit tests
> onto an existing spec.
> * It is immediately clear to implementors if a spec change requires a
> corresponding change in their implementation.
> * It is clear to spec authors if they are introducing a change that
> unexpectedly breaks existing implementations.
>
> I think the last points in particular are compelling. Although this seems
> like more work upfront, the feedback I have heard from specs like HTML and
> DOM is that requiring tests along with spec changes speeds up the overall
> specification process, because it's more likely that changes stick and
> aren't reverted later due to unforeseen compat issues. It also means that
> specification changes are more likely to reflected in implementations
> quickly (since there's an obvious failing test to fix), so it helps avoid
> the situation where a change is made and then implementations do nothing
> (because they simply didn't notice) and eventually it is reverted (or,
> worse, some implementations notice and make a fix and others don't, and
> clients end up relying on the behavioural differences).
>
> I strongly recommend that we adopt this requirement or writing tests for
> all post-L1 spec changes.
>
>