Re: Review of tests upstreamed by implementors

On Wednesday, March 20, 2013 at 10:29 PM, Rebecca Hauck wrote:
> And you're probably right about
> implementors not running all or many of the W3C tests. The quandary is
> that implementors seem to want to only take tests (not authored by them)
> that are blessed by the W3C and the W3C wants implementors to bless the
> tests by running them.

 One of my goals is to make sure that we automatically run the tests on a number of different user agents both on the master branch and on incoming pull requests.

This should help identify a number of issues upfront, and circumvent the quandary you mentioned above.
> Perhaps part of "working with implementors" is 1)
> resetting their expectations on the state of W3C tests

I'd rather try to improve that state. I'm convinced that this can be done by making it easy to run the tests, look at the results, report issues and send patches.
> and 2) persuading
> them to regularly accept and run all tests in whatever state with the
> understanding that if something goes wrong, the onus is on the implementor
> to (re)review the test and resolve the inconsistency wherever it may lie.

Fixing bugs is everyone's responsibility. The more visibility the test suites have, the higher the incentive to do so. 
> I'm personally thrilled that we're having this conversation. Even though
> the solution that emerges from this may not be immediately obvious, this
> thread gets to the very essence of What W3C Tests are For.

Me too. :)

--tobie

Received on Wednesday, 20 March 2013 21:46:02 UTC