Re: Review of tests upstreamed by implementors

Hi Dirk,

On 20/03/2013 22:46 , Dirk Pranke wrote:
> While you're right about the quandary, I'm not sure that putting the
> onus on the implementor to review and resolve issues with any and all
> tests is the way to go. I suspect most implementors will share my
> concern that we might be being asked to run completely arbitrary tests
> without regard to their quality or value.
>
> I believe a challenge lies in building up some level of trust between
> implementors and the testing groups of the W3C that the "submitted" (and
> even "approved") w3c tests aren't wasting our time :).

I can certainly see where you're coming from. But we have a bit of a 
chicken and egg problem here. If implementers don't really use the 
tests, that decreases the value of the test suite and creates a dearth 
of feedback, which makes it harder to produce high-quality test (and to 
find motivation to produce tests in the first place), which in turn cuts 
off any incentive you may have to run the tests in the first place.

We need to prime the iteration loop in which implementations benefit 
from off-the-shelf testing and the testing people benefit from 
implementers' feedback.

I won't lie to you, there are thousands of tests in the test suite and I 
would be surprised if the first few times an implementer will run the 
full suite, it won't be in part a (short-term) waste of time. But I can 
equally promise that the issues coming out of it will be a high priority 
item for us to fix and that in short order it will switch from a 
time-waster to a time-saver. We're all in the same boat making the Web a 
better place and all that, we just need to Frankenstein the symbiont.

> Another challenge lies in making it absolutely painless to pull new
> tests and run them. Fixing this one first (which is what I believe James
> was largely alluding to) will go a long way to building up the trust
> (and make it easier to build it up further).

I actually think that that's the first challenge to get the virtuous 
cycle kick-started. In the first runs it's likely that there will be 
problems, the least we can do is to make performing a test run as easy 
to set up as possible.

Any requirements you can provide in this area would be extremely helpful.

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Thursday, 21 March 2013 09:45:19 UTC