RE: Conversion of MS CSS 2.1 tests to reftests

> -----Original Message-----
> From: public-css-testsuite-request@w3.org [mailto:public-css-testsuite-
> request@w3.org] On Behalf Of Sylvain Galineau
> Sent: Tuesday, September 21, 2010 10:34 AM
> To: Boris Zbarsky
> Cc: public-css-testsuite@w3.org
> Subject: RE: Conversion of MS CSS 2.1 tests to reftests
> 
> Like it or not, that's what it takes us. There are long series of tests that do not
> take anywhere near 11s/testcase. Others do take longer. No slave labor
> necessary but not a leisurely 9-to-5 schedule either. And yes, we do have
> someone who knows it backwards and forwards and can likely do it faster
> than most. So let's be very conservative and call it 5 days. That is still
> substantially less, imo, than building an automation system, testing it,
> converting part - or all ? - of the testcases to reftests (or whatever input
> format the automation expects), finding any mistakes from that process etc.
> If that's your preferred course of action, great. But unless  you had a
> significant headstart before the Oslo meeting then 10/15 was a completely
> unrealistic deadline. Is it too much to ask for you to acknowledge this and tell
> the WG:
> "This is the way we're going to proceed. Therefore we can't submit an
> Implementation Report by 10/15; it won't be ready before X" ? If you think
> you can still make 10/15 that'd be good to know too. From all the arguing I'm
> not getting that vibe though.
> 
> Understand that I'm not really questioning why or how you want to do it. All
> I'm asking is that you offer you best estimate as to how long it's going to take
> you.
> 
> As for mistakes, they are certainly possible in a manual process. Since we run
> it across browser, we can certainly compare your results with ours. That
> should shake out a lot of false positives/negatives on both sides.
> 
> -----Original Message-----
> From: Boris Zbarsky [mailto:bzbarsky@MIT.EDU]
> Sent: Tuesday, September 21, 2010 10:21 AM
> To: Sylvain Galineau
> Cc: public-css-testsuite@w3.org
> Subject: Re: Conversion of MS CSS 2.1 tests to reftests
> 
> On 9/21/10 11:21 AM, Sylvain Galineau wrote:
> > Running the test suite takes about 3 days.
> 
> The HTML 4.01 version of the test suite has 7989 tests in it.  3 work days is
> 86400 seconds.  So you're saying that the test suite can be run at 11 seconds
> per test without taking any breaks, right?  Without slave labor?  And without
> making mistakes?
> 
> Or did you mean that if you task enough people with doing it you can run the
> test suite in 3 days?

Please note, that I ran the entire suite for the first time last summer and it took me three days of interrupted time (NOT non-interrupted time). I just now ran 20 tests from the HTML suite and it took me 24 seconds.

I am not saying this is typical, necessarily, and when you hit a failure it certainly adds time, but I think that looking at an 11 second average seems very high in practice.

-John

> 
> Fwiw, I just tried running a few of the tests, and I think 30 seconds per test is
> a good estimate for the simple ones (that's how long they take me to run
> given the network lag, etc); the more complicated ones need more time than
> that to just read.  That gives me a lower bound of about 8 person-days,
> assuming 100% efficiency.  I'd be really surprised if someone can run the test
> suite for more than an hour or two straight without starting to make
> mistakes, though, so you either need to have redundancy or a lot more
> people doing a bit at a time...
> 
> That's just the HTML version of the test suite, note.
> 
> -Boris
> 
>

Received on Tuesday, 21 September 2010 17:42:16 UTC