- From: Ian Hickson <ian@hixie.ch>
- Date: Thu, 5 Jun 2014 20:27:59 +0000 (UTC)
- To: Robin Berjon <robin@w3.org>
- cc: public-html@w3.org
On Wed, 4 Jun 2014, Robin Berjon wrote: > > All the results (careful, it's a bit big) > http://w3c.github.io/test-results/html/all.html Looking at this, I think the numbers here are really misleading. There aren't 100,000 tests, there's more like 1000 tests, but they have lots of assertions. If we were to count assertions rather than tests, then the old SVG test suite with its 275 tests could be counted as millions of tests, since you could assert each pixel in turn when comparing the reference image to the test image. Same with the CSS test suite; it had several thousand tests, but each one tested dozens of things and their interactions, so it could be described as millions of tests also. Specifically, for example, this file alone accounts for 17,760 tests: http://www.w3c-test.org/html/dom/reflection-embedded.html ...and it's only testing a few requirements in the spec (most of them actually in WebIDL). It's a great test, but it's literally more than 17% of the entire test suite, that's how small the test suite is. In this context, the HTML test suite as it stands today, while a great effort, is nowhere near ready to be used for determining whether the spec as a whole is interoperably implemented, and is nowhere near enough to actually prove that we have interoperable browsers in general. Entire sections lack any sort of serious testing, for example there's nothing testing the Navigation algorithm, which is probably the spec's most important algorithm, there's nothing testing image loading, there's barely anything testing tables, the tests for things like form controls are testing barely more than the Web API... (BTW, I picked one test at random when writing the above: http://w3c-test.org/html/browsers/history/the-history-interface/003.html ...and it has a serious error. There's no initPopStateEvent(). That was the first test I looked at; I didn't review more in detail.) -- Ian Hickson U+1047E )\._.,--....,'``. fL http://ln.hixie.ch/ U+263A /, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Received on Thursday, 5 June 2014 20:28:23 UTC