Re: Report is getting slow

On 02/11/2011 12:07 AM, Philippe Le Hegaret wrote:

> I'm also thinking to remove the percentage numbers except for Canvas, ie
> we would only generate those once we reach a certain number of tests for
> each feature.

If we have to display results at all at this stage, I strongly support 
this. I think reporting percentages of tests passed is dubious at best 
and in the case of known-incomplete testsuites outright harmful. I don't 
think this information does anything to help foster interoperability, 
but it is very tempting for people to use as a tl;dr summary without 
regard for the fact that the coverage is incomplete, different tests can 
be more or less difficult to fix, or represent more or less serious bugs.

I would also advise against using absolute number as a metric for 
completeness. Instead a section should be considered complete if it can 
be shown that most of the normative statements in that section are 
covered, at least in a simple way. Philip has done this for the <canvas> 
tests I believe. Obviously this doesn't account for the desire to test 
features in combination. Such tests are obviously good, but simple 
mathematics dictates that we will never have enough tests to make a 
complete test of all combinations.

Received on Friday, 11 February 2011 11:13:08 UTC