RE: Conversion of MS CSS 2.1 tests to reftests

Like it or not, that's what it takes us. There are long series of tests that do not take anywhere near 11s/testcase. Others do take longer. No slave labor necessary but 
not a leisurely 9-to-5 schedule either. And yes, we do have someone who knows it backwards and forwards and can likely do it faster than most. So let's be very 
conservative and call it 5 days. That is still substantially less, imo, than building an automation system, testing it, converting part - or all ? - of the testcases to reftests 
(or whatever input format the automation expects), finding any mistakes from that process etc. If that's your preferred course of action, great. But unless  you had
a significant headstart before the Oslo meeting then 10/15 was a completely unrealistic deadline. Is it too much to ask for you to acknowledge this and tell the WG: 
"This is the way we're going to proceed. Therefore we can't submit an Implementation Report by 10/15; it won't be ready before X" ? If you think you can still
make 10/15 that'd be good to know too. From all the arguing I'm not getting that vibe though. 

Understand that I'm not really questioning why or how you want to do it. All I'm asking is that you offer you best estimate as to how long it's going to take you.

As for mistakes, they are certainly possible in a manual process. Since we run it across browser, we can certainly compare your results with ours. That should
shake out a lot of false positives/negatives on both sides.

-----Original Message-----
From: Boris Zbarsky [mailto:bzbarsky@MIT.EDU] 
Sent: Tuesday, September 21, 2010 10:21 AM
To: Sylvain Galineau
Cc: public-css-testsuite@w3.org
Subject: Re: Conversion of MS CSS 2.1 tests to reftests

On 9/21/10 11:21 AM, Sylvain Galineau wrote:
> Running the test suite takes about 3 days.

The HTML 4.01 version of the test suite has 7989 tests in it.  3 work days is 86400 seconds.  So you're saying that the test suite can be run at 11 seconds per test without taking any breaks, right?  Without slave labor?  And without making mistakes?

Or did you mean that if you task enough people with doing it you can run the test suite in 3 days?

Fwiw, I just tried running a few of the tests, and I think 30 seconds per test is a good estimate for the simple ones (that's how long they take me to run given the network lag, etc); the more complicated ones need more time than that to just read.  That gives me a lower bound of about 8 person-days, assuming 100% efficiency.  I'd be really surprised if someone can run the test suite for more than an hour or two straight without starting to make mistakes, though, so you either need to have redundancy or a lot more people doing a bit at a time...

That's just the HTML version of the test suite, note.

-Boris

Received on Tuesday, 21 September 2010 17:35:04 UTC