- From: Sylvain Galineau <sylvaing@microsoft.com>
- Date: Tue, 21 Sep 2010 17:36:27 +0000
- To: Geoffrey Sneddon <gsneddon@opera.com>
- CC: Anne van Kesteren <annevk@opera.com>, "L. David Baron" <dbaron@dbaron.org>, John Jansen <John.Jansen@microsoft.com>, fantasai <fantasai.lists@inkedblade.net>, Arron Eicholz <Arron.Eicholz@microsoft.com>, "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>
It will be a sunk cost once it is built and running. Right now, you're saying it won't be complete in time unless you get help so the cost is not sunk *yet*. Running the test suite and publishing the results is a sunk cost too once it's done and over with. While it's ideal to submit an IR for all the platforms you support, I wonder if all three are strictly necessary for the purpose of IR submission ? -----Original Message----- From: Geoffrey Sneddon [mailto:gsneddon@opera.com] Sent: Tuesday, September 21, 2010 10:14 AM To: Sylvain Galineau Cc: Anne van Kesteren; L. David Baron; John Jansen; fantasai; Arron Eicholz; public-css-testsuite@w3.org Subject: Re: Conversion of MS CSS 2.1 tests to reftests On 21/09/10 16:21, Sylvain Galineau wrote: > Running the test suite takes about 3 days. I very much doubt that > building an automation system, testing it and converting thousands of > testcases takes less time. Building such an automation system and testing it is AFAIK a sunk cost for all the major browser vendors apart from Microsoft. And it's not just three days to run the test-case once: it's three days per platform to run in once. Windows, Mac, and Linux is nine days to run the tests. We can't realistically run the entire testsuite often (on Desktop, excluding other products for now) if it takes nine days to run, as it's just not an economically feasible use of QA resources, and it will result in testcases being regressed. Yes, sure, Opera's automation system /can/ cope with just taking screenshots, but then you have to label the screenshots, which means three days per product per platform, which simply does not scale to the number of products we have, especially seeming things can and do change (a default stylesheet change, a library update changing AA of fonts, etc.), as well as the almost unavoidable randomness within such a system requiring screenshots to be labeled more often than would be desired. -- Geoffrey Sneddon - Opera Software <http://gsnedders.com> <http://opera.com>
Received on Tuesday, 21 September 2010 17:37:07 UTC