Re: HTML test results

On Wed, 4 Jun 2014, Robin Berjon wrote:
>
>     All the results (careful, it's a bit big)
>     http://w3c.github.io/test-results/html/all.html

Looking at this, I think the numbers here are really misleading. There 
aren't 100,000 tests, there's more like 1000 tests, but they have lots of 
assertions. If we were to count assertions rather than tests, then the old 
SVG test suite with its 275 tests could be counted as millions of tests, 
since you could assert each pixel in turn when comparing the reference 
image to the test image. Same with the CSS test suite; it had several 
thousand tests, but each one tested dozens of things and their 
interactions, so it could be described as millions of tests also.

Specifically, for example, this file alone accounts for 17,760 tests:

   http://www.w3c-test.org/html/dom/reflection-embedded.html

...and it's only testing a few requirements in the spec (most of them 
actually in WebIDL). It's a great test, but it's literally more than 17% 
of the entire test suite, that's how small the test suite is.

In this context, the HTML test suite as it stands today, while a great 
effort, is nowhere near ready to be used for determining whether the spec 
as a whole is interoperably implemented, and is nowhere near enough to 
actually prove that we have interoperable browsers in general. Entire 
sections lack any sort of serious testing, for example there's nothing 
testing the Navigation algorithm, which is probably the spec's most 
important algorithm, there's nothing testing image loading, there's barely 
anything testing tables, the tests for things like form controls are 
testing barely more than the Web API...


(BTW, I picked one test at random when writing the above:

   http://w3c-test.org/html/browsers/history/the-history-interface/003.html

...and it has a serious error. There's no initPopStateEvent(). That was 
the first test I looked at; I didn't review more in detail.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Received on Thursday, 5 June 2014 20:28:23 UTC