Re: Vendor harnesses

On May 10, 2011, at 2:25 AM, James Graham wrote:

> On 05/10/2011 12:55 AM, Linss, Peter wrote:
> 
>> I get that vendors are all using their own home-grown systems, and
>> integrating those with the official W3C test suites is a problem
>> we've been wrestling with for years already in CSS. Personally, I'd
>> like to see the W3C test systems be robust enough that they cover the
>> internal needs of the vendors and they can simply leverage them
>> directly (with perhaps reasonable extension hooks for truly one-off
>> uses). Ideally the differences between the official test systems and
>> the internal testing systems should approach zero over time. But I'm
>> well aware that's a long way off and we have to have a way to get
>> there from here.
> 
> I think that should be an explicit non-goal.
> 
> I believe trying to create test runner that everyone is happy to use for 
> all their internal needs it will lead to impossible-to-reconcile 
> conflicting requirements. I think the goal instead should be to produce 
> things that integrate as well as possible into a variety of existing 
> setups. For example Opera really like running our tests over the scope 
> protocol [1]. Expecting other vendors to adopt that approach is clearly 
> a non-starter, but we wouldn't want to replace our existing systems with 
> ones less well adapted to our requirements.
> 
> There are also more mundane issues to consider. For example "never rely 
> on an external server" is the first commandment of getting reliable 
> testing. So we will always have to make some adaptations to ensure that 
> testcases never do that.
> 
> [1] http://my.opera.com/dragonfly/blog/scope-protocol-specification

I don't agree that synergy between vendor's existing internal testing systems and the W3C's testing system is a "non-goal".

I do agree that we shouldn't be spending any time _right now_ trying to design or build a test harness such that we expect all vendors to drop their own solutions and adopt it. 

First and foremost is the issue of the tests themselves. In the CSS testing effort we found that pretty much every vendor has their own test suite and testing system. Getting people to write tests is hard, getting them to write them _twice_ is virtually impossible. The W3C needs to be aware of existing test suites and testing systems and goal #1 of the W3C's test system needs to be the ability of vendors to take the W3C test suites and drop them into their internal testing systems with as little effort as possible, and vice versa. This way we all have one common corpus of tests we all use and we maximize the availability of tests. I don't really think there's any disagreement on this point.

Now, when it come to the harness, I obviously didn't spend a whole lot of time worrying about the internal testing procedures of every vendor for the CSS harness, but the harness does have a mechanism to exchange result data with external systems (implementation report importing and result exporting) and this has proven to be very useful.

When I talk about my goal here (and I did say it was a personal goal), what I mean is that over time, I'd like to see the W3C system evolve to the point that it can serve the same needs as what vendors need for their internal testing. Ideally, if a new vendor comes to the scene down the road, they'll be able to simply adopt the W3C testing system rather than roll their own. And while I'm not expecting existing vendors to drop their own systems and switch to the W3C's, I do want to see the systems converge, so that at least where there are overlaps in functionality, the same tools can be used. 

I don't want to see the W3C testing system try to adapt to every vendor's proprietary testing hooks, but it would be good if the W3C's testing system was extensible so that vendors could write their own adapters to connect their testing hooks to the W3C harness, for example. Even better, would be for browser vendors to agree on a standardized testing API so that any browser can integrate into a standard testing harness.

Let me give a concrete example, with Firefox, you can make a special build of the browser that can automatically compare reference tests and gather results. Wouldn't it be useful if that build could run the W3C's testing harness directly and submit results there?

Again, though, for me this is a long term goal. I'm not proposing that the W3C spend the next two years trying to figure out how to do this and we block all progress while while trying to reconcile conflicting requirements. We need to be building systems now that meet the needs of the W3C first. This is just something I'm keeping in the back of my mind and will strive towards incrementally as opportunities present themselves.