W3C home > Mailing lists > Public > public-css-testsuite@w3.org > September 2010

Re: Conversion of MS CSS 2.1 tests to reftests

From: Geoffrey Sneddon <gsneddon@opera.com>
Date: Tue, 21 Sep 2010 18:14:22 +0100
Message-ID: <4C98E7EE.8030707@opera.com>
To: Sylvain Galineau <sylvaing@microsoft.com>
CC: Anne van Kesteren <annevk@opera.com>, "L. David Baron" <dbaron@dbaron.org>, John Jansen <John.Jansen@microsoft.com>, fantasai <fantasai.lists@inkedblade.net>, Arron Eicholz <Arron.Eicholz@microsoft.com>, "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>
On 21/09/10 16:21, Sylvain Galineau wrote:
> Running the test suite takes about 3 days. I very much doubt that
> building an automation system, testing it and converting thousands
> of testcases takes less time.

Building such an automation system and testing it is AFAIK a sunk cost 
for all the major browser vendors apart from Microsoft.

And it's not just three days to run the test-case once: it's three days 
per platform to run in once. Windows, Mac, and Linux is nine days to run 
the tests. We can't realistically run the entire testsuite often (on 
Desktop, excluding other products for now) if it takes nine days to run, 
as it's just not an economically feasible use of QA resources, and it 
will result in testcases being regressed.

Yes, sure, Opera's automation system /can/ cope with just taking 
screenshots, but then you have to label the screenshots, which means 
three days per product per platform, which simply does not scale to the 
number of products we have, especially seeming things can and do change 
(a default stylesheet change, a library update changing AA of fonts, 
etc.), as well as the almost unavoidable randomness within such a system 
requiring screenshots to be labeled more often than would be desired.

-- 
Geoffrey Sneddon  Opera Software
<http://gsnedders.com>
<http://opera.com>
Received on Tuesday, 21 September 2010 17:15:17 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 21 September 2010 17:15:29 GMT