Vendor harnesses

On 05/10/2011 12:55 AM, Linss, Peter wrote:

> I get that vendors are all using their own home-grown systems, and
> integrating those with the official W3C test suites is a problem
> we've been wrestling with for years already in CSS. Personally, I'd
> like to see the W3C test systems be robust enough that they cover the
> internal needs of the vendors and they can simply leverage them
> directly (with perhaps reasonable extension hooks for truly one-off
> uses). Ideally the differences between the official test systems and
> the internal testing systems should approach zero over time. But I'm
> well aware that's a long way off and we have to have a way to get
> there from here.

I think that should be an explicit non-goal.

I believe trying to create test runner that everyone is happy to use for 
all their internal needs it will lead to impossible-to-reconcile 
conflicting requirements. I think the goal instead should be to produce 
things that integrate as well as possible into a variety of existing 
setups. For example Opera really like running our tests over the scope 
protocol [1]. Expecting other vendors to adopt that approach is clearly 
a non-starter, but we wouldn't want to replace our existing systems with 
ones less well adapted to our requirements.

There are also more mundane issues to consider. For example "never rely 
on an external server" is the first commandment of getting reliable 
testing. So we will always have to make some adaptations to ensure that 
testcases never do that.

[1] http://my.opera.com/dragonfly/blog/scope-protocol-specification

Received on Tuesday, 10 May 2011 09:25:32 UTC