RE: Conversion of CSS 2.1 RC1 tests to reftests

Hi All,

For what it's worth: we have a [X]HTML to PDF converter, I took the
tests (9952 XHT files) converted them all to PDF and start checking.

I have now done 4408 in 3 hours (4 times 3/4 hour), and approved 3948 of
them, the remaining 460 needs to be re-checked. 

I agree that this re-checking will take longer, but the overall speed is
very high as you can see.  


Best Regards,
Robert Stam
 
TallComponents
Follow us @ twitter: http://twitter.com/tallcomponents


> -----Original Message-----
> From: public-css-testsuite-request@w3.org
[mailto:public-css-testsuite-
> request@w3.org] On Behalf Of John Jansen
> Sent: Tuesday, September 21, 2010 7:42 PM
> To: Sylvain Galineau; Boris Zbarsky
> Cc: public-css-testsuite@w3.org
> Subject: RE: Conversion of MS CSS 2.1 tests to reftests
> 
> 
> > -----Original Message-----
> > From: public-css-testsuite-request@w3.org
[mailto:public-css-testsuite-
> > request@w3.org] On Behalf Of Sylvain Galineau
> > Sent: Tuesday, September 21, 2010 10:34 AM
> > To: Boris Zbarsky
> > Cc: public-css-testsuite@w3.org
> > Subject: RE: Conversion of MS CSS 2.1 tests to reftests
> >
> > Like it or not, that's what it takes us. There are long series of
tests
> that do not
> > take anywhere near 11s/testcase. Others do take longer. No slave
labor
> > necessary but not a leisurely 9-to-5 schedule either. And yes, we do
> have
> > someone who knows it backwards and forwards and can likely do it
faster
> > than most. So let's be very conservative and call it 5 days. That is
> still
> > substantially less, imo, than building an automation system, testing
it,
> > converting part - or all ? - of the testcases to reftests (or
whatever
> input
> > format the automation expects), finding any mistakes from that
process
> etc.
> > If that's your preferred course of action, great. But unless  you
had a
> > significant headstart before the Oslo meeting then 10/15 was a
> completely
> > unrealistic deadline. Is it too much to ask for you to acknowledge
this
> and tell
> > the WG:
> > "This is the way we're going to proceed. Therefore we can't submit
an
> > Implementation Report by 10/15; it won't be ready before X" ? If you
> think
> > you can still make 10/15 that'd be good to know too. From all the
> arguing I'm
> > not getting that vibe though.
> >
> > Understand that I'm not really questioning why or how you want to do
it.
> All
> > I'm asking is that you offer you best estimate as to how long it's
going
> to take
> > you.
> >
> > As for mistakes, they are certainly possible in a manual process.
Since
> we run
> > it across browser, we can certainly compare your results with ours.
That
> > should shake out a lot of false positives/negatives on both sides.
> >
> > -----Original Message-----
> > From: Boris Zbarsky [mailto:bzbarsky@MIT.EDU]
> > Sent: Tuesday, September 21, 2010 10:21 AM
> > To: Sylvain Galineau
> > Cc: public-css-testsuite@w3.org
> > Subject: Re: Conversion of MS CSS 2.1 tests to reftests
> >
> > On 9/21/10 11:21 AM, Sylvain Galineau wrote:
> > > Running the test suite takes about 3 days.
> >
> > The HTML 4.01 version of the test suite has 7989 tests in it.  3
work
> days is
> > 86400 seconds.  So you're saying that the test suite can be run at
11
> seconds
> > per test without taking any breaks, right?  Without slave labor?
And
> without
> > making mistakes?
> >
> > Or did you mean that if you task enough people with doing it you can
run
> the
> > test suite in 3 days?
> 
> Please note, that I ran the entire suite for the first time last
summer
> and it took me three days of interrupted time (NOT non-interrupted
time).
> I just now ran 20 tests from the HTML suite and it took me 24 seconds.
> 
> I am not saying this is typical, necessarily, and when you hit a
failure
> it certainly adds time, but I think that looking at an 11 second
average
> seems very high in practice.
> 
> -John
> 
> >
> > Fwiw, I just tried running a few of the tests, and I think 30
seconds
> per test is
> > a good estimate for the simple ones (that's how long they take me to
run
> > given the network lag, etc); the more complicated ones need more
time
> than
> > that to just read.  That gives me a lower bound of about 8
person-days,
> > assuming 100% efficiency.  I'd be really surprised if someone can
run
> the test
> > suite for more than an hour or two straight without starting to make
> > mistakes, though, so you either need to have redundancy or a lot
more
> > people doing a bit at a time...
> >
> > That's just the HTML version of the test suite, note.
> >
> > -Boris
> >
> >
>

Received on Tuesday, 21 September 2010 18:06:49 UTC