HTML5 Test Development Estimation

Interested HTML5 Test Developers -


My name is Kevin Kershaw and I work as an engineering manager at CableLabs in Denver, CO.  If you're not currently aware of us, CableLabs develops standards, software, and test tools as well as basic doing basic R&D and testing activities on behalf of Cable TV operators - primarily in North America.  We have a small group of engineers here who are interested in providing additional test coverage for HTML5 features to the W3C, in particular, we want to develop tests for features around media presentation that are of critical concern for the operator community.  Our engineers have spent some time looking into the set of existing HTML5 tests and the associated test harness and we're at the point now where I'd like to get some advice from the broader HTML5 test community.



First off, our team is specifically looking at building tests for designated subsections of HTML5 section 4.8.  We originally identified video, audio, track, and media elements in our scope but added the source element and  Dimension Attributes because of the tight coupling we see between these.  Also, we've excluded some Media Element subsections (e.g. MediaControl) for our initial work.  We started a "bottom-up" analysis of the target sections, working to identify what looked to us to be "testable" requirements in the spec.  The subsections of the spec itself divide up pretty nicely by individual paragraphs.  That is, each paragraph usually lists one or more potential test conditions.  We did some basic tabulation of requirements within each paragraph to come up with a count of potential tests.  I've included the spreadsheet we constructed to assist this process in this email.  That sheet is pretty self-explanatory but if you have questions, I'm more than happy to answer.  Our analysis was done by several different engineers, each of whom had slightly different ideas about how to count "tests" but the goal here was to produce an approximation, not a perfectly accurate list.



One interesting result of this work was that the number of tests we came up with differed substantially from the estimates prepared recently by the W3C team for their "Open Web Platform Test Suite - Coverage Analysis and Cost Estimate".  We identified about 517 tests for the subsections that were in scope for us.  This compares to about 2611 tests expected by the W3C team for those same sections - a difference factor of about 5.  In no way do I mean to suggest that our estimates are better than those of the W3C team.  We just took a different approach and came up with a different number.  But, from the perspective of applying resources to developing the needed tests, this represents a pretty significant cost difference and since I'm looking at funding from within CableLabs to get this work done, my management would like to know the likely cost of the effort.  They really want to know if we're developing on the order of 500 tests or 2500 tests and I'm looking to validate my assumptions and analysis results.



I've attached a workbook that contains the "bottoms-up" spreadsheet we used to count up the anticipated number of tests and another small worksheet where I compared our test counts with those generated earlier by the W3C team..   Here's a short list of assumptions we made as we worked through the problem



*         We excluded tests of the IDL from both the W3C and CableLabs estimates under the assumption that the IDLHarness will generate IDL tests automatically.

*         We accounted for some tests around algorithms but believe that many algorithm steps, especially intermediate steps, do not require separate tests.

*         We subtracted out the number of existing, approved tests in the GitHub repository that were associated with our target sections in order to come up with a count of "remaining" tests to be developed.

*         We assumed that a suitable test harness and driver will be available to run the set of developed tests.  I understand there's significant work to be done on that infrastructure but that's not part of this little exercise.



My request to the community at this point is to ask for your thoughts on the validity of our estimation approach and whether we've missed some substantial aspect of the problem in our analysis.



We appreciate any feedback you might have and look forward to working with you to improve the coverage of the HTML5 test tools.



Thanks & regards,



Kevin Kershaw

CableLabs