Re: Conversion of MS CSS 2.1 tests to reftests

On 21/09/10 16:21, Sylvain Galineau wrote:
> Running the test suite takes about 3 days. I very much doubt that
> building an automation system, testing it and converting thousands
> of testcases takes less time.

Building such an automation system and testing it is AFAIK a sunk cost 
for all the major browser vendors apart from Microsoft.

And it's not just three days to run the test-case once: it's three days 
per platform to run in once. Windows, Mac, and Linux is nine days to run 
the tests. We can't realistically run the entire testsuite often (on 
Desktop, excluding other products for now) if it takes nine days to run, 
as it's just not an economically feasible use of QA resources, and it 
will result in testcases being regressed.

Yes, sure, Opera's automation system /can/ cope with just taking 
screenshots, but then you have to label the screenshots, which means 
three days per product per platform, which simply does not scale to the 
number of products we have, especially seeming things can and do change 
(a default stylesheet change, a library update changing AA of fonts, 
etc.), as well as the almost unavoidable randomness within such a system 
requiring screenshots to be labeled more often than would be desired.

Geoffrey Sneddon  Opera Software

Received on Tuesday, 21 September 2010 17:15:17 UTC