Re: Getting browser vendors running and submitting tests

On Thu, Oct 29, 2015 at 1:44 PM, Chris Lilley <chris@w3.org> wrote:

> Hello Geoffrey,
>
> Wednesday, October 28, 2015, 5:21:33 PM, you wrote:
> > * We should get rid of metadata, at least in the common case,
>
> The crucial bit of metadata is a link to *what section of the spec is
> actually tested*.
>
> If someone writing a test cannot provide a link to  what is actually
> being tested, the test is at best worthless and at worst, damaging.
> But for the common case, a test author does know, they have the
> relevant spec open as they write the test, so dropping in that link
> is trivial to do.
>
> Certainly there are other metadata items which are optional, but this
> one is simple to provide and adds a lot of value.


The majority of tests browser vendors write do not have any sort of
metadata in them—as James says, browser vendors will and have point-blank
refused to submit tests that require any such metadata. If we want our test
suite to actually be useful and contain the majority of tests browsers run,
we need to get to a point where browser vendors are willing to submit
tests. Even if you somehow convince browser vendors to include links in all
future tests they write, this still leaves us without the tens of thousands
of tests browser vendors already have.

As has been pointed out on numerous occasions (though I think many F2F
rather in the summaries I've written here), we can make this metadata
implicit in the directory structure, and that's something people *are*
willing to work with (because it doesn't require modifying, quite possibly
by hand, and reviewing, existing tests).

/Geoffrey