Re: Getting browser vendors running and submitting tests

> On 29 Oct 2015, at 16:57, Geoffrey Sneddon <me@gsnedders.com> wrote:
> 
> 
> 
> On Thu, Oct 29, 2015 at 4:23 PM, Florian Rivoal <florian@rivoal.net <mailto:florian@rivoal.net>> wrote:
> 
> > On 29 Oct 2015, at 11:48, James Graham <james@hoppipolla.co.uk <mailto:james@hoppipolla.co.uk>> wrote:
> >
> > On 29/10/15 11:37, fantasai wrote:
> >
> >> I disagree with this. I think we should reduce the metadata,
> >> but there are some things (e.g. spec section associations)
> >> that we need to keep.
> >
> > FWIW the counterpoint to this is that people have and will point-blank refuse to submit tests when there are requirements for metadata beyond what is strictly needed to make the tests run.
> >
> > I understand why this additional metadata is nice to have particularly when you come back to tests later, but requiring it will cause people to not upstream tests that they otherwise would have. I don't have a great solution for you, but consider if they are ways to make more of the metadata implicit in e.g. the directory structure, file naming, <title> element, etc.
> 
> I think that a test that has neither a pointer to the spec section it is testing nor an explicit assertion is close to being unusable, unreviewable, and unmaintainable, and that we don't loose much if it is not being submitted.
> 
> We should reduce the amount of metadata required, but not to 0.
> 
> That implies that many tests that browsers have are "unusable, unreviewable, and unmaintainable", so I don't think that's a true statement! After all, it seems likely that browsers would've changed what they're doing if it were so bad.

Let me qualify that statement a little. "unusable, unreviewable, and unmaintainable" to a third party.

When a browser vendor writes a test together with new implementation, or targeting an existing part of their implementation, they have this metadata (not the whole set of what we require today, but at least the "what are you testing" information), at least in their heads, probably in their bug tracker as well.

That makes the test useful to them, since they know what they are testing.

I challenge the idea that there are tests in the browsers vendors's private respositories that have been checked in independently of the code (not just in a separate commit, but on a branch that makes it impossible to know what code they relate to), and have no information anywhere (not in the commit message, not in the test, not in the bug tracker) about what these tests are.

If we allow tests to be shared entirely without information, this is the situation we would be in. Other browser vendors, receiving such a test, would suddenly have a  failing test (or a few hundred) in the CI test suite, with no indication about that that is about.

The overhead of figuring out what such a test does is far higher than the overhead of writing it down when you already know it. It is quite likely that, for the receiving party, such tests would indeed be "unusable, unreviewable, and unmaintainable".

 - Florian