Re: Requiring tests for spec changes

On 04/09/17 15:13, Andreas Tolfsen wrote:
> Large sections of the spec is currently untested.  Does this mean it
> is a requirement for the change to come with a full test suite for
> that section?  I can see this would hinder or discourage rightful
> improvements, if making the change also means one has to spend
> another week filling out the missing test gaps.

I think the rule would be that the tests should be sufficient to 
demonstrate that an implementation matched the specific changes 
introduced by the PR. So this wouldn't require backfilling tests for the 
same feature with no bearing on the PR.

> Would tests submitted in downstream repositories, i.e.
> mozilla-central, be acceptable?  As someone working directly on
> implementations, it is often more convenient to rely on Mozilla’s
> CI when writing tests because they can immediately be annotated with
> their expected pass/failure metadata.

I think the rule would be that you have to link to the test PR from the 
spec PR. So any process that made that possible would be fine. The 
specific issue of adding gecko metadata shouldn't dictate your choices 
here because that is added by the import script and it's already easy to 
see how gecko behaves using |wpt run|. If you are making a 
spec/test/implementation change all at once, it might make sense to make 
a m-c patch  with the implementation change and tests, and link from the 
spec change.

> This brings me on to my final concern, that upstream test changes
> take a long time before reaching downstream browser vendor
> repositories.  When I make a specification change I often try to
> ensure the implementation doesn’t lag too far behind.  If I can
> submit the tests to m-c when making the spec change this won’t a
> problem, but waiting for a WPT update would mean implementations
> will lag behind somewhat, considering we can’t land changes to
> geckodriver without sufficient test coverage.

This is a Mozilla-specific issue that should go away soon one we have 
what literally no one is calling "quantum sync" i.e. the ability to run 
wpt syncs automatically and  continuously rather than in batches like today.

Received on Monday, 4 September 2017 14:49:38 UTC