RE: Conversion of MS CSS 2.1 tests to reftests

> -----Original Message-----
> From: Geoffrey Sneddon [mailto:gsneddon@opera.com]
> Sent: Tuesday, September 28, 2010 3:33 AM
> To: Sylvain Galineau
> Cc: Anne van Kesteren; L. David Baron; John Jansen; fantasai; Arron
> Eicholz; public-css-testsuite@w3.org
> Subject: Re: Conversion of MS CSS 2.1 tests to reftests
> 
> On 21/09/10 18:36, Sylvain Galineau wrote:
> > It will be a sunk cost once it is built and running.
> 
> The reftest runner is a sunk cost for everyone apart from MS. I expect
> it'd take around 1w full-time to convert half the testsuite to reftests.

Cool. Are you doing that ? It sounded like you need the help of others for
that part. 

> Given five browsers (i.e., the number of browser vendors in the WG) it
> would, at the currently quoted time, take 15 (working) days to run. If
> we spend 5 days automating stuff and get it down to 7.5 days to run the
> testsuite for all vendors, we've made a net gain. And that's just when
> running the testsuite once, and I doubt we're just going to do it once.

Sure, but some vendors have already spent time running the test suite at
this stage (Apple,Google) so it may not be a net gain for the purpose of
this first IR. 

> 
> > Right now, you're saying it won't be complete in time unless you get
> > help so the cost is not sunk *yet*.
> 
> I've not changed from what I said at the F2F: Opera can submit an IR
> for
> the HTML 4.01 copy of the testsuite within a month of the RC being
> published (i.e., by 17th October), as well as for the nonHTML tests.
> (Depending on resource availability, this IR may or may not include
> tests flagged with interact, active, and userstyle.)

Very glad to hear it ! Frankly, it wasn't clear whether much would actually 
happen from the way Anne described Opera's position. 

> 
> This does /not/ mean we don't want to invest time into automating the
> tests.
> 
> > Running the test suite and publishing the results is a sunk cost too
> > once it's done and over with.
> 
> It's a sunk cost only if you do it purely for the sake of writing an IR.
> If you want to use the tests across multiple platforms for regression
> tracking, that's only one instance of the cost.

Sure. If you didn't already have such a system then of course it's a good
thing to have.

> 
> > While it's ideal to submit an IR for all the platforms you support,
> > I wonder if all three are strictly necessary for the purpose of IR
> > submission ?
> 
> I wonder what affect us solely submitting a report for Windows will be:
> there are features such as font-weight which aren't supported on
> Windows
> except for in IE9 due to GDI limitations. I guess IE9/Safari (which I
> presume Apple will submit IR for on OS X) should pass them, at least...

I didn't imply you should submit only one platform or that it should be
Windows. Two is always better than one. I just questioned whether you 
did need all three. I don't believe you do.

Received on Tuesday, 28 September 2010 23:15:22 UTC