W3C home > Mailing lists > Public > public-browser-tools-testing@w3.org > July to September 2017

Re: Requiring tests for spec changes

From: Simon Stewart <simon.m.stewart@gmail.com>
Date: Mon, 4 Sep 2017 11:36:38 +0100
Message-ID: <CAOrAhYGs3HCJJZNMOWst1E_GNa36QYjgZ5BmJiUQ+mtzJF1REA@mail.gmail.com>
To: James Graham <james@hoppipolla.co.uk>
Cc: "public-browser-tools-testing@w3.org" <public-browser-tools-testing@w3.org>
I'm in favour of this. It'll also help us in the case where the meaning of
the spec text isn't clear --- at least there's an alternative description
of the expected behaviour.


On Mon, Sep 4, 2017 at 11:31 AM, James Graham <james@hoppipolla.co.uk>

> Now that L1 is mostly done, it's time to start thinking about how we can
> learn from the experience and improve the process for future work.
> One thing that is being adopted by other specs to great success is to
> require that spec changes come with an accompanying test change, where
> possible (i.e. excluding things that don't affect the normative
> requirements). This has a number of advantages:
> * We have confidence that the spec has at least basic tests for everything.
> * We don't get stuck in the current situation of trying to retro-fit tests
> onto an existing spec.
> * It is immediately clear to implementors if a spec change requires a
> corresponding change in their implementation.
> * It is clear to spec authors if they are introducing a change that
> unexpectedly breaks existing implementations.
> I think the last points in particular are compelling. Although this seems
> like more work upfront, the feedback I have heard from specs like HTML and
> DOM is that requiring tests along with spec changes speeds up the overall
> specification process, because it's more likely that changes stick and
> aren't reverted later due to unforeseen compat issues. It also means that
> specification changes are more likely to reflected in implementations
> quickly (since there's an obvious failing test to fix), so it helps avoid
> the situation where a change is made and then implementations do nothing
> (because they simply didn't notice) and eventually it is reverted (or,
> worse, some implementations notice and make a fix and others don't, and
> clients end up relying on the behavioural differences).
> I strongly recommend that we adopt this requirement or writing tests for
> all post-L1 spec changes.
Received on Monday, 4 September 2017 10:37:05 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:09:56 UTC