Re: Automating testcases

On 09/05/2010 11:41 AM, Geoffrey Sneddon wrote:
> On 05/09/10 19:44, Bjoern Hoehrmann wrote:
>> * Geoffrey Sneddon wrote:
>>> However, the first large set of tests that it makes sense to automate
>>> are tests submitted by Microsoft: there are 400–500 tests for which the
>>> reference is effectively just "<p>Test passes if there is no red visible
>>> on the page.</p>". This is the largest collection of tests in the
>>> testsuite with a single reference.
>>>
>>> If we (i.e., the browser vendors apart from MS represented at the
>>> meeting, Apple, Google, Mozilla, and Opera) wish to convert them to
>>> reftests, how should we go about this?
>>
>> Checking that a browser's rendering of a document contains no red is a
>> one-liner (IECapt | ImageMagick's identify | grep for "#FF0000 red"),
>> so would the task be to make sure a script can easily identify which of
>> the tests can be run this way and the exact pass condition (in addition
>> to "no red" there is also "some green", "only green" and so on)? That
>> should only be a matter of collecting all the prose from the tests and
>> then identify the right patterns.
>
> Well, how to actually do the automation isn't the question (the obvious
> solution being to follow the already established policy of using
> reftests), but rather how to do it within the policies about touching
> tests you don't own, when the majority of people active within the
> testing group want them automated but the owners do not.

I'm sure if you wrote a reftest manifest for Microsoft's tests and
sent it to Arron to look over and asked if you could check it in,
he'd say sure, go ahead. There's no conflict of interest in improving
our test suites here. There's just a difference of interest wrt what
is most important to spend resources on improving.

You're free to write patches to improve the tests (or convert them to
reftests or whatever), but shouldn't check them in without the owner's
permission. This rule is about respecting the ownership of the tests.
But I mean that in the "module ownership" meaning of ownership, not
the "property ownership" meaning of ownership.

If there's actually a conflict of interest, then you can come complain
to me. I'm the module owner for the test suite overall. But, like Anne,
I really don't forsee a problem here.

~fantasai

Received on Tuesday, 7 September 2010 18:57:23 UTC