Re: Review of tests upstreamed by implementors

On Wed, Mar 20, 2013 at 2:29 PM, Rebecca Hauck <rhauck@adobe.com> wrote:

>
> >
> >It is my opinion that we should be putting considerably more effort into
> >working with all implementors to ensure that they are actually running
> >the
> >tests that we are collecting.
>
> I doubt anyone disagrees with this. And you're probably right about
> implementors not running all or many of the W3C tests. The quandary is
> that implementors seem to want to only take tests (not authored by them)
> that are blessed by the W3C and the W3C wants implementors to bless the
> tests by running them. Perhaps part of "working with implementors" is 1)
> resetting their expectations on the state of W3C tests and 2) persuading
> them to regularly accept and run all tests in whatever state with the
> understanding that if something goes wrong, the onus is on the implementor
> to (re)review the test and resolve the inconsistency wherever it may lie.
>
>
While you're right about the quandary, I'm not sure that putting the onus
on the implementor to review and resolve issues with any and all tests is
the way to go. I suspect most implementors will share my concern that we
might be being asked to run completely arbitrary tests without regard to
their quality or value.

I believe a challenge lies in building up some level of trust between
implementors and the testing groups of the W3C that the "submitted" (and
even "approved") w3c tests aren't wasting our time :).

Another challenge lies in making it absolutely painless to pull new tests
and run them. Fixing this one first (which is what I believe James was
largely alluding to) will go a long way to building up the trust (and make
it easier to build it up further).

Note that in this scenario, implementors' must help out here. We
(implementors) can hardly blame the W3C for problems if we've put no effort
in on our side.

-- Dirk