RE: Conversion of MS CSS 2.1 tests to reftests

You can call it a 'blanket assertion' to justify disagreeing with it, Boris. This is the amount of time it takes Arron, John and others
to run it. What could seem weird, actually, is that this is pretty much what it usually takes whether the person is very fast - like Arron -
or does it for the first time (John, not long ago). Because, well, they're not 'slave labor', there are interruptions and no one can really
do this for 8-10 hours straight. But the bottom line is that the repeated average in our experience is substantially *less* than 11 seconds 
per test. It's not a blanket assertion, it's just a measured fact. If you need more time than them, again, that's fine by me. But it's still highly
unlikely to take you more time than building a reliable system that does the same job unattended.

As dbaron has often noted, there is in fact a lot of repetition in the test suite e.g. check for the same thing for top/right/bottom/left
variations of a property. So the first testcase may take you 8 seconds to load/read but the other three take 2 seconds each to check, if that. 
There are other series that are far longer than four. So the number of testcases that take 11 seconds is actually a small subset of the total. 
Most take a lot less.

To be clear, reviews are also performed using a simple iframe and some script to do previous/next and register pass/fail that took 
very little time to produce. It sounds like Tab has already set up such a system so you don't even have to build it yourself. Robert Stam
uses a different method that works for him.

So I would suggest trying it on a reasonable subset so 'blanket assertions' and disagreements can at least be called out on an empirical basis.

I understand that this is only your opinion. But do understand that I am rather embarrassed that we are having arguments about the number 
of seconds in 3 working days less than a month before a deadline that was agreed to nearly a month ago. Never mind some of the claims on this 
thread that this process is being 'rushed'. We started submitting the test suite in early 2008. Testcases were given for review to WG members in 
March of 2009 in Tokyo. Mozilla completed the submission of their own testcases - including your own, if I recall - back in...January 2010 ? The WG 
has been spending 100% of its telcon  time on CSS2.1 issues for *months*. Elika has been contracting for Microsoft to work on the test suite for 
months as well. 

Unless you have reason to believe the WG had nothing else to do or talk about, the conclusion of this process as well as its timing should not be a 
surprise or an issue for anyone. And if some members of the WG would rather spend 3+ weeks building an automated system than 3 days running 
a bunch of testcases  - or 9 days for three platforms... - that is their choice and their responsibility. Not mine. 

But as responsible members of a working group with a collective deadline, I expect them to be open and accountable about any delay from that deadline 
that may result from their chosen course of action. I don't think that is a crazy expectation. Although I'm starting to realize I might be wrong.

-----Original Message-----
From: public-css-testsuite-request@w3.org [mailto:public-css-testsuite-request@w3.org] On Behalf Of Boris Zbarsky
Sent: Tuesday, September 21, 2010 11:19 AM
To: Sylvain Galineau
Cc: public-css-testsuite@w3.org
Subject: Re: Conversion of MS CSS 2.1 tests to reftests

On 9/21/10 1:34 PM, Sylvain Galineau wrote:
> So let's be very
> conservative and call it 5 days. That is still substantially less, 
> imo, than building an automation system, testing it, converting part - or all ? - of the testcases to reftests (or whatever input format the automation expects), finding any mistakes from that process etc.

That last part, I agree with.  I just disagree with the blanket assertion that it's possible to run the test suite and analyze the failures in 3 days for one person in its current state.

Also, note that my comments here are my personal comments, NOT representing Mozilla in any way.  If there is an official Mozilla statement here, it'll be clearly labeled as such, I assure you....

-Boris

Received on Tuesday, 21 September 2010 19:32:14 UTC