Re: Conversion of MS CSS 2.1 tests to reftests

On 21/09/10 18:36, Sylvain Galineau wrote:
> It will be a sunk cost once it is built and running.

The reftest runner is a sunk cost for everyone apart from MS. I expect
it'd take around 1w full-time to convert half the testsuite to reftests.
Given five browsers (i.e., the number of browser vendors in the WG) it
would, at the currently quoted time, take 15 (working) days to run. If
we spend 5 days automating stuff and get it down to 7.5 days to run the
testsuite for all vendors, we've made a net gain. And that's just when
running the testsuite once, and I doubt we're just going to do it once.

> Right now, you're saying it won't be complete in time unless you get
> help so the cost is not sunk *yet*.

I've not changed from what I said at the F2F: Opera can submit an IR for
the HTML 4.01 copy of the testsuite within a month of the RC being
published (i.e., by 17th October), as well as for the nonHTML tests.
(Depending on resource availability, this IR may or may not include
tests flagged with interact, active, and userstyle.)

This does /not/ mean we don't want to invest time into automating the tests.

> Running the test suite and publishing the results is a sunk cost too
> once it's done and over with.

It's a sunk cost only if you do it purely for the sake of writing an IR.
If you want to use the tests across multiple platforms for regression
tracking, that's only one instance of the cost.

> While it's ideal to submit an IR for all the platforms you support,
> I wonder if all three are strictly necessary for the purpose of IR
> submission ?

I wonder what affect us solely submitting a report for Windows will be: 
there are features such as font-weight which aren't supported on Windows
except for in IE9 due to GDI limitations. I guess IE9/Safari (which I
presume Apple will submit IR for on OS X) should pass them, at least…

-- 
Geoffrey Sneddon — Opera Software
<http://gsnedders.com>
<http://opera.com>

Received on Tuesday, 28 September 2010 13:54:59 UTC