Re: when automerging of tests goes awry

Thanks for raising this issue, Patrick!

I've commented https://github.com/w3c/navigation-timing/issues/59 to verify
that this is a prove-it-then-spec-it situation, which is not uncommon but
hasn't been a problem until now. We had a similar situation in
https://github.com/whatwg/html/issues/2249#issuecomment-278847830

Whenever possible, I think we should try to merge the spec changes close to
the time that the impl+test changes are committed and exported. But however
hard we try, we'll have some cases where we deliberately want to try
shipping something first.

I see two options with different tradeoffs:

   - We put tests like this outside of web-platform-tests, even if that
   means making a copy. Once the spec change is made, hopefully someone is
   around to consolidate the differences and get the coverage into
   web-platform-tests. This requires a bit of discipline and I think we'll
   keep having mishaps, even if they become fewer.
   - We accept such tentative changes inside of web-platform-tests, but
   either with a special test name like -tentative.html, or in a specific
   top-level directory for this. It still requires discipline, but when a
   mishap is discovered, it's possible to fix within web-platform-tests by
   reverting and duplicating. That's not to say we *want* others to do it,
   but it seems nicer for everyone to not just be reverting.

I prefer the second option. Thoughts?

On Tue, Mar 7, 2017 at 8:41 AM Patrick Kettner <patket@microsoft.com> wrote:

> EHLO folks,
>
> As we are all aware, the Chrome team recently began automatically
> upstreaming their layout tests with their new sync system. Todd Reifsteck,
> a coworker on perf and networking, brought an issue to my attention that
> feels like incorrect behavior that resulting because of it. A change is
> being proposed <https://github.com/w3c/navigation-timing/issues/59> to
> the navigation timing spec, which Chrome is wanting to implement. They have
> already implemented the feature, tests and all. As a result when it was
> merged into Chromium, tests included, it overrode the existing tests in
> WPT
> <https://github.com/w3c/web-platform-tests/commit/d24a86b138b2074f32a6f2bbd6ca5de445259dd3>
> via the sync. Now WPT is testing for behavior that is a breaking change
> from a previous level of the spec, before the change has even been made to
> the spec.
>
>
>
> This feels wrong to me, but I was interested in what everyone else thought
> the behavior should be.
>
>
>
> patrick
>
>
>