RE: Conversion of MS CSS 2.1 tests to reftests

> Also, I think it's a bad idea to behave as if work on the testsuite will stop with the first IRs are published.
No one suggested you 'behave' that way. Only that given a ~3 day cost for publish that first set of IR, a two-months deadline seemed quite reasonable.
If  you *also* have time to solve the larger problem of automating regression testing for everyone, great. But the latter should either help you and others
submit the first batch of IRs, or have no impact on their submission i.e. it shouldn't prevent it. Some of the arguments offered implied that such work
could or even should prevent IRs from being submitted if that's what it took. I'm glad that's not the case.

-----Original Message-----
From: Geoffrey Sneddon [mailto:gsneddon@opera.com] 
Sent: Wednesday, September 29, 2010 9:35 AM
To: Sylvain Galineau
Cc: Anne van Kesteren; L. David Baron; John Jansen; fantasai; Arron Eicholz; public-css-testsuite@w3.org
Subject: Re: Conversion of MS CSS 2.1 tests to reftests

On 29/09/10 00:14, Sylvain Galineau wrote:
>
>> -----Original Message----- From: Geoffrey Sneddon 
>> [mailto:gsneddon@opera.com] Sent: Tuesday, September 28, 2010 3:33 AM 
>> To: Sylvain Galineau Cc: Anne van Kesteren; L. David Baron; John 
>> Jansen; fantasai; Arron Eicholz; public-css-testsuite@w3.org
>> Subject: Re: Conversion of MS CSS 2.1 tests to reftests
>>
>> On 21/09/10 18:36, Sylvain Galineau wrote:
>>> It will be a sunk cost once it is built and running.
>>
>> The reftest runner is a sunk cost for everyone apart from MS. I 
>> expect it'd take around 1w full-time to convert half the testsuite to 
>> reftests.
>
> Cool. Are you doing that ? It sounded like you need the help of others 
> for that part.

As I said before, we will provide resources as part of a larger combined effort. (That said, I may do some more in my own time, but that's
unpredictable.)

(For those interested: 6 references can automate 10% of the testsuite,
100 can automate 50% of the testsuite. The majority of the other 50% of the testsuite currently has unique screenshots, so would require an almost equal number of references.)

>> Given five browsers (i.e., the number of browser vendors in the WG) 
>> it would, at the currently quoted time, take 15 (working) days to 
>> run. If we spend 5 days automating stuff and get it down to 7.5 days 
>> to run the testsuite for all vendors, we've made a net gain.
>> And that's just when running the testsuite once, and I doubt we're 
>> just going to do it once.
>
> Sure, but some vendors have already spent time running the test suite 
> at this stage (Apple,Google) so it may not be a net gain for the 
> purpose of this first IR.

So have we (as stated before, we have the majority of the testsuite within our regression tracking system, running it by screenshot matching), yet there is an unavoidable long-term cost to running them in such a way. I'm not saying this has to be done in time for the first IR, but it would be better to get the cost of automating it fully out of the way before running it multiple times.

Also, I think it's a bad idea to behave as if work on the testsuite will stop with the first IRs are published. I'd hope vendors (apart from Opera and MS) will include the testsuite within their regression tests so that the quality of implementations does not deteriorate over time, and both ourselves and the other vendors would much rather for this the testsuite contained a far greater number of reftests for this.

>>> While it's ideal to submit an IR for all the platforms you support, 
>>> I wonder if all three are strictly necessary for the purpose of IR 
>>> submission ?
>>
>> I wonder what affect us solely submitting a report for Windows will
>> be: there are features such as font-weight which aren't supported on 
>> Windows except for in IE9 due to GDI limitations. I guess IE9/Safari 
>> (which I presume Apple will submit IR for on OS X) should pass them, 
>> at least...
>
> I didn't imply you should submit only one platform or that it should 
> be Windows. Two is always better than one. I just questioned whether 
> you did need all three. I don't believe you do.

I never said you did imply such a thing (if anything, the implication of my comment is that an IR we submit with the testsuite as it is currently would be on Windows only, which is true), I was merely wondering what effect it will have on the spec's progression to REC.

--
Geoffrey Sneddon - Opera Software
<http://gsnedders.com>
<http://opera.com>

Received on Wednesday, 29 September 2010 23:05:37 UTC