W3C home > Mailing lists > Public > public-html-testsuite@w3.org > February 2011

Re: Report is getting slow

From: James Graham <jgraham@opera.com>
Date: Fri, 11 Feb 2011 11:01:10 +0100
Message-ID: <4D5508E6.7030301@opera.com>
To: Philippe Le Hegaret <plh@w3.org>
CC: Kris Krueger <krisk@microsoft.com>, "public-html-testsuite@w3.org" <public-html-testsuite@w3.org>
On 02/11/2011 12:07 AM, Philippe Le Hegaret wrote:

> I'm also thinking to remove the percentage numbers except for Canvas, ie
> we would only generate those once we reach a certain number of tests for
> each feature.

If we have to display results at all at this stage, I strongly support 
this. I think reporting percentages of tests passed is dubious at best 
and in the case of known-incomplete testsuites outright harmful. I don't 
think this information does anything to help foster interoperability, 
but it is very tempting for people to use as a tl;dr summary without 
regard for the fact that the coverage is incomplete, different tests can 
be more or less difficult to fix, or represent more or less serious bugs.

I would also advise against using absolute number as a metric for 
completeness. Instead a section should be considered complete if it can 
be shown that most of the normative statements in that section are 
covered, at least in a simple way. Philip has done this for the <canvas> 
tests I believe. Obviously this doesn't account for the desire to test 
features in combination. Such tests are obviously good, but simple 
mathematics dictates that we will never have enough tests to make a 
complete test of all combinations.
Received on Friday, 11 February 2011 11:13:08 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Friday, 11 February 2011 11:13:09 GMT