- From: Geoffrey Sneddon <gsneddon@opera.com>
- Date: Tue, 21 Sep 2010 18:14:22 +0100
- To: Sylvain Galineau <sylvaing@microsoft.com>
- CC: Anne van Kesteren <annevk@opera.com>, "L. David Baron" <dbaron@dbaron.org>, John Jansen <John.Jansen@microsoft.com>, fantasai <fantasai.lists@inkedblade.net>, Arron Eicholz <Arron.Eicholz@microsoft.com>, "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>
On 21/09/10 16:21, Sylvain Galineau wrote: > Running the test suite takes about 3 days. I very much doubt that > building an automation system, testing it and converting thousands > of testcases takes less time. Building such an automation system and testing it is AFAIK a sunk cost for all the major browser vendors apart from Microsoft. And it's not just three days to run the test-case once: it's three days per platform to run in once. Windows, Mac, and Linux is nine days to run the tests. We can't realistically run the entire testsuite often (on Desktop, excluding other products for now) if it takes nine days to run, as it's just not an economically feasible use of QA resources, and it will result in testcases being regressed. Yes, sure, Opera's automation system /can/ cope with just taking screenshots, but then you have to label the screenshots, which means three days per product per platform, which simply does not scale to the number of products we have, especially seeming things can and do change (a default stylesheet change, a library update changing AA of fonts, etc.), as well as the almost unavoidable randomness within such a system requiring screenshots to be labeled more often than would be desired. -- Geoffrey Sneddon — Opera Software <http://gsnedders.com> <http://opera.com>
Received on Tuesday, 21 September 2010 17:15:17 UTC