Re: Rough first draft of Level 0

On Apr 1, 2012, at 21:32 , James Graham wrote:
> I don't think that will work. There is simply too much stuff that is untested or poorly tested at present. On the other hand it would be very useful for this document to either point to the existing public testsuite for features or note the lack of such a testsuite as a substantial problem that prevents browsers interoperating on untested features. It would be even better if we could get people that are not already contributing to testsuites to do so.

I would be very happy to capture that information as issues in the document. I would appreciate some help in doing so (I know you have a GH account :).

If we get a decent shopping list that can provide us with a good set of testing gaps it will make it a lot easier to talk to various people who could dedicate resources to this. For instance (example totally off the top of my head), "We need people to write tests for these N aspects of canvas so that the Web platform is competitive for these types of games" can get us much farther than "It would be nice, you know, if, like, we contributed tests to the Web. Dude, you know, karma and stuff." One of those translates into clear business requirements and it's a lot easier to hang money off those.

> Unless there are substantially more ringmark-related tests that I have seen, that unfortunately seems like it will fall well short of testing anything to the level needed to demonstrate high quality, interoperable, implementations. Although the number of tests needed for any given feature depends strongly on the size of the feature, one typically needs ~hundreds of tests for good coverage. In some cases it may be even more e.g. the HTML5 parser has a few thousand tests, the Opera drag and drop testsuite -- which we should release -- has in excess of 1000 tests, the DOM Range testsuite produces something like 60,000 results, and so on. To get an idea of the scale of the problem, Wilhelm (chair of the Browser Testing interest group) estimated something like 1000000 tests will be needed to properly cover HTML5. I think that's not an unreasonable figure.

I think we all agree on this, but the answer shouldn't be "we need millions of tests so we should give up!" rather it ought to be "we need millions of tests, so we should organise to target the ones that will have the greatest impact on interoperability first".

We need to keep in mind that even if we need a million tests, the first thousand may do more for interoperability than the following ten thousand, in turn than the following hundred thousand. Especially if we do it right.

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Thursday, 12 April 2012 13:44:29 UTC