RE: HTML5 Test Development Estimation

Mike -

Thanks for your feedback.  Yours are all good observations and I couldn't agree more on the role of experience.  I think our team has good experience developing testing for complex systems, including some messy platforms with "intertwingled" parts.  But no question, writing tests for HTML will be new for our small team.  I'm just working to organize the tasks we need to cover in order to start making an effective contribution.  Then we can apply experience gained to the next chunk of work we bite off.

Regards,

Kevin K.


-----Original Message-----
From: Michael[tm] Smith [mailto:mike@w3.org] 
Sent: Tuesday, August 06, 2013 2:26 AM
To: Kevin Kershaw
Cc: public-test-infra; James Graham; Tobie Langel; Takashi Hayakawa; Brian Otte; Nishant Shah
Subject: Re: HTML5 Test Development Estimation

Kevin Kershaw <K.Kershaw@cablelabs.com>, 2013-08-05 16:33 +0000:

> 4)  WRT "what is a test" and the time it takes to write one.   I'm not
> married to the 8 hours and I'd be happy to change it if subsequent 
> experience warrents.  The estimate is my approximation based on my 
> experience w/ this kind of spec compliance testing.  For me, it's the 
> aggregate of the time you spend figuring out what the spec is saying,

It seems like the time required to do that depends quite highly on the level of experience that the people writing the tests have with the HTML spec.

If they're at the level of not being familiar at all with the spec yet and have never written tests for it, then that step of figuring out what the spec is saying is going to take them a lot of time. But after, say, a week of writing tests and getting familiar with the spec, that figuring-out- what-the-spec is saying time will go way down. And after two weeks it'll go down even further. So I think that needs to be figured in into any estimates.

To me at least, the skills needed for people to write tests well for HTML5 and the other Web-platform specs seem different than skills needed for people who write tests for other kinds of systems -- among other things because the platform is messy and the parts are intertwingled with each other all over the place, to a higher degree than in other systems.

So, somebody new who's thrown into writing tests for the Web platform is going to have a relatively steep learning curve, and a relatively large amount of their test-writing time will be spent just getting up to speed.

To me that suggests (among other things) that rather than trying to take a bunch of new people and have them write a large amount of Web-platform tests in a short amount of time, you're way better off doing it with fewer people and taking more time: A small set of people who spend significant time first getting familiar with the specs (HTML5 and the other specs it normatively depends on) and writing a few tests, and then start writing a lot more tests in earnest after they've gotten over the initial learning hump.

I think that's true of course to some degree of any kind of test-writing work for a large system. But I think it's something that's much more acutely true for Web-platform test writing.

  --Mike

--
Michael[tm] Smith http://people.w3.org/mike

Received on Tuesday, 6 August 2013 19:13:59 UTC