Re: HTML5 Test Development Estimation

Kevin Kershaw <K.Kershaw@cablelabs.com>, 2013-08-05 16:33 +0000:

> 4)  WRT "what is a test" and the time it takes to write one.   I'm not
> married to the 8 hours and I'd be happy to change it if subsequent
> experience warrents.  The estimate is my approximation based on my
> experience w/ this kind of spec compliance testing.  For me, it's the
> aggregate of the time you spend figuring out what the spec is saying,

It seems like the time required to do that depends quite highly on the level
of experience that the people writing the tests have with the HTML spec.

If they're at the level of not being familiar at all with the spec yet and
have never written tests for it, then that step of figuring out what the
spec is saying is going to take them a lot of time. But after, say, a week
of writing tests and getting familiar with the spec, that figuring-out-
what-the-spec is saying time will go way down. And after two weeks it'll go
down even further. So I think that needs to be figured in into any estimates.

To me at least, the skills needed for people to write tests well for HTML5
and the other Web-platform specs seem different than skills needed for
people who write tests for other kinds of systems -- among other things
because the platform is messy and the parts are intertwingled with each
other all over the place, to a higher degree than in other systems.

So, somebody new who's thrown into writing tests for the Web platform is
going to have a relatively steep learning curve, and a relatively large
amount of their test-writing time will be spent just getting up to speed.

To me that suggests (among other things) that rather than trying to take a
bunch of new people and have them write a large amount of Web-platform tests
in a short amount of time, you're way better off doing it with fewer people
and taking more time: A small set of people who spend significant time first
getting familiar with the specs (HTML5 and the other specs it normatively
depends on) and writing a few tests, and then start writing a lot more
tests in earnest after they've gotten over the initial learning hump.

I think that's true of course to some degree of any kind of test-writing
work for a large system. But I think it's something that's much more
acutely true for Web-platform test writing.

  --Mike

-- 
Michael[tm] Smith http://people.w3.org/mike

Received on Tuesday, 6 August 2013 08:26:37 UTC