Re: HTML test results

On 05/06/14 21:27, Ian Hickson wrote:
> On Wed, 4 Jun 2014, Robin Berjon wrote:
>>
>>     All the results (careful, it's a bit big)
>>     http://w3c.github.io/test-results/html/all.html
> 
> Looking at this, I think the numbers here are really misleading. There 
> aren't 100,000 tests, there's more like 1000 tests, but they have lots of 
> assertions. 

That's not a sensible way to read these tests. Counting files doesn't
make sense because you have genuinely separate tests in a single file,
not just separate asserts. The number Robin gave is much more accurate.

If we were to count assertions rather than tests, then the old
> SVG test suite with its 275 tests could be counted as millions of tests, 
> since you could assert each pixel in turn when comparing the reference 
> image to the test image. Same with the CSS test suite; it had several 
> thousand tests, but each one tested dozens of things and their 
> interactions, so it could be described as millions of tests also.

I think that's an absurd way to interpret those tests, because the state
of different pixels in a single test is correlated.

I agree with the general point that counting tests isn't, on its own, a
very useful exercise.

> In this context, the HTML test suite as it stands today, while a great 
> effort, is nowhere near ready to be used for determining whether the spec 
> as a whole is interoperably implemented, and is nowhere near enough to 
> actually prove that we have interoperable browsers in general.

I don't think it's a secret that there are many areas in which browsers
are not as interoperable as we would like. You don't need a testsuite to
tell you that. I don't think that should halt progress on the spec.

> (BTW, I picked one test at random when writing the above:
> 
>    http://w3c-test.org/html/browsers/history/the-history-interface/003.html
> 
> ...and it has a serious error. There's no initPopStateEvent(). That was 
> the first test I looked at; I didn't review more in detail.)

Presumably there was one at some time, since it is implemented in at
least Gecko and Presto. But a bug report or, better still, a patch,
would be welcome.

Received on Thursday, 5 June 2014 21:05:51 UTC