Re: Coverage analysis

On 03/07/2013 03:58 PM, Robin Berjon wrote:
> It has a lot of tests :) Those were written for html5lib. In fact,
> according to James, if I understand correctly, I've undercounted those
> by a factor of *three* because that specific sub testsuite runs three
> different batches (I'm guessing the same parsing but in different
> contexts) depending on the query string.

Yes, these tests either run with the input string as a data URI, or with 
the input string injected via document.write all at once or with the 
input string injected via document.write one character at a time.

In terms of the coverage of the parsing specification this difference 
doesn't mean much. In terms of implementations this difference is huge 
(or can be, depending on implementation strategy). We found a whole raft 
of bugs with the single-character-at-a-time document.write tests that we 
didn't see with the other types.

In general it should be realised that "number of tests per X" is only a 
proxy for the quality of the testsuite, and can be a bad proxy e.g. 
adding a hundred tests for <font color> with different colours isn't 
adding any value.

A better proxy would look actual implementations and see how well the 
testsuite is covering the code. This is theoretically possible to do, at 
least with open source implementations, but it is obviously a bit of 
work to make gcov, or similar, enabled builds and interpret the results 
in a sensible way.

Received on Thursday, 7 March 2013 15:20:40 UTC