Re: Rough first draft of Level 0

On Sun, 1 Apr 2012, Tobie Langel wrote:

> On 4/1/12 4:17 PM, "Arthur Barstow" <Art.Barstow@nokia.com> wrote:
>
>> Where is the criteria used to add a feature to the spec?
>
> It's right here:
> http://www.w3.org/community/coremob/wiki/Specs/Coremob_Level_0
>
>> It seems like for this spec to actually be useful, a criterion for adding
>> a feature to the spec should be a publicly available comprehensive test
>> suite for the feature.

I don't think that will work. There is simply too much stuff that is 
untested or poorly tested at present. On the other hand it would be very 
useful for this document to either point to the existing public testsuite 
for features or note the lack of such a testsuite as a substantial problem 
that prevents browsers interoperating on untested features. It would be 
even better if we could get people that are not already contributing to 
testsuites to do so.

> That's a very good point. Testing is at the core of this CG, as expressed
> in its charter. Facebook is going to be donating a whole bunch of tests
> which were recently written for Ringmark (http://rng.io).

Unless there are substantially more ringmark-related tests that I have 
seen, that unfortunately seems like it will fall well short of testing 
anything to the level needed to demonstrate high quality, interoperable, 
implementations. Although the number of tests needed for any given feature 
depends strongly on the size of the feature, one typically needs ~hundreds 
of tests for good coverage. In some cases it may be even more e.g. the 
HTML5 parser has a few thousand tests, the Opera drag and drop testsuite 
-- which we should release -- has in excess of 1000 tests, the DOM 
Range testsuite produces something like 60,000 results, and so on. To 
get an idea of the scale of the problem, Wilhelm (chair of the Browser 
Testing interest group) estimated something like 1000000 tests will be 
needed to properly cover HTML5. I think that's not an unreasonable figure.

All of this isn't supposed to suggest that getting the ringmark tests 
won't be appreacited; it very much will. But we should be clear on what 
"testing" means to browser vendors. "Testing" is about ensuring the 
quality and interoperability of implementations. That is a very different 
goal to the goal that the ringmark tests seem to aspire to which is to 
test for a basic level of support for certian features and for the 
avoidance of some rather specific known bugs. To give a concrete example 
of a difference, a comprehensive testsuite might include a test that a 
browser fails whilst still having excellent support for a feature. For 
this reason scoring such testsuites as pass -- meaning 100% of tests pass 
-- or fail -- meaning < 100% of tests pass -- isn't approprite.

Received on Sunday, 1 April 2012 19:32:53 UTC