Requiring tests for spec changes

Also sprach James Graham:

> One thing that is being adopted by other specs to great success
> is to require that spec changes come with an accompanying test
> change, where possible (i.e. excluding things that don't affect
> the normative requirements). This has a number of advantages:
> 
> * We have confidence that the spec has at least basic tests
> for everything. * We don't get stuck in the current situation
> of trying to retro-fit tests onto an existing spec. * It is
> immediately clear to implementors if a spec change requires a
> corresponding change in their implementation. * It is clear to
> spec authors if they are introducing a change that unexpectedly
> breaks existing implementations.

It’s hard to be opposed to a process that will increase test
coverage for changes or new features, however I have a couple of
practical questions.

Large sections of the spec is currently untested.  Does this mean it
is a requirement for the change to come with a full test suite for
that section?  I can see this would hinder or discourage rightful
improvements, if making the change also means one has to spend
another week filling out the missing test gaps.

I would very much be fine with an iterative approach where one
tries to add a specific test for the change one is making, without
worrying too much about the existing lack of coverage.

Would tests submitted in downstream repositories, i.e.
mozilla-central, be acceptable?  As someone working directly on
implementations, it is often more convenient to rely on Mozilla’s
CI when writing tests because they can immediately be annotated with
their expected pass/failure metadata.

This brings me on to my final concern, that upstream test changes
take a long time before reaching downstream browser vendor
repositories.  When I make a specification change I often try to
ensure the implementation doesn’t lag too far behind.  If I can
submit the tests to m-c when making the spec change this won’t a
problem, but waiting for a WPT update would mean implementations
will lag behind somewhat, considering we can’t land changes to
geckodriver without sufficient test coverage.

Received on Monday, 4 September 2017 14:13:54 UTC