Re: Conversion of MS CSS 2.1 tests to reftests

On 29/09/10 00:14, Sylvain Galineau wrote:
>
>> -----Original Message----- From: Geoffrey Sneddon
>> [mailto:gsneddon@opera.com] Sent: Tuesday, September 28, 2010 3:33
>> AM To: Sylvain Galineau Cc: Anne van Kesteren; L. David Baron; John
>> Jansen; fantasai; Arron Eicholz; public-css-testsuite@w3.org
>> Subject: Re: Conversion of MS CSS 2.1 tests to reftests
>>
>> On 21/09/10 18:36, Sylvain Galineau wrote:
>>> It will be a sunk cost once it is built and running.
>>
>> The reftest runner is a sunk cost for everyone apart from MS. I
>> expect it'd take around 1w full-time to convert half the testsuite
>> to reftests.
>
> Cool. Are you doing that ? It sounded like you need the help of
> others for that part.

As I said before, we will provide resources as part of a larger combined
effort. (That said, I may do some more in my own time, but that's
unpredictable.)

(For those interested: 6 references can automate 10% of the testsuite,
100 can automate 50% of the testsuite. The majority of the other 50% of
the testsuite currently has unique screenshots, so would require an
almost equal number of references.)

>> Given five browsers (i.e., the number of browser vendors in the WG)
>> it would, at the currently quoted time, take 15 (working) days to
>> run. If we spend 5 days automating stuff and get it down to 7.5
>> days to run the testsuite for all vendors, we've made a net gain.
>> And that's just when running the testsuite once, and I doubt we're
>> just going to do it once.
>
> Sure, but some vendors have already spent time running the test suite
> at this stage (Apple,Google) so it may not be a net gain for the
> purpose of this first IR.

So have we (as stated before, we have the majority of the testsuite
within our regression tracking system, running it by screenshot
matching), yet there is an unavoidable long-term cost to running them in
such a way. I'm not saying this has to be done in time for the first IR,
but it would be better to get the cost of automating it fully out of the
way before running it multiple times.

Also, I think it's a bad idea to behave as if work on the testsuite will
stop with the first IRs are published. I'd hope vendors (apart from
Opera and MS) will include the testsuite within their regression tests
so that the quality of implementations does not deteriorate over time,
and both ourselves and the other vendors would much rather for this the
testsuite contained a far greater number of reftests for this.

>>> While it's ideal to submit an IR for all the platforms you
>>> support, I wonder if all three are strictly necessary for the
>>> purpose of IR submission ?
>>
>> I wonder what affect us solely submitting a report for Windows will
>> be: there are features such as font-weight which aren't supported
>> on Windows except for in IE9 due to GDI limitations. I guess
>> IE9/Safari (which I presume Apple will submit IR for on OS X)
>> should pass them, at least...
>
> I didn't imply you should submit only one platform or that it should
> be Windows. Two is always better than one. I just questioned whether
> you did need all three. I don't believe you do.

I never said you did imply such a thing (if anything, the implication of
my comment is that an IR we submit with the testsuite as it is currently
would be on Windows only, which is true), I was merely wondering what
effect it will have on the spec's progression to REC.

-- 
Geoffrey Sneddon — Opera Software
<http://gsnedders.com>
<http://opera.com>

Received on Wednesday, 29 September 2010 23:00:16 UTC