Re: Shall we shelve Level Zero?

On Wed, 2 May 2012, Jet Villegas wrote:

> I would prefer that the rings represent features like ring 0 == text, 
> ring 1 == images, ring 2 == JavaScript, ring 3 == audio, ring 4 == 
> video...

So I think that makes a lot of sense for a testsuite. However I don't 
think the primary goal of this group is to produce a testsuite, although I 
certianly encourage the production of tests as part of the work. It seems 
to me that this group exists to set an agenda; that is to get the relevant 
stakeholders from the worlds of content production, browser development 
and web standards together to put together a rough priority order on 
nascent web technologies so that everyone is pulling in the same 
direction; browser vendors are implementing the things that authors want 
the most, standards committees are putting the most effort into 
stabalising the things that are being implemeneted, and authors are 
desigining sites using cross-browser technologies rather than a 
smorgasboard of UA-specific hacks.

Given this outlook; the ringed approach makes a great deal more sense 
because at long as we agree that the next thing that we want authors to be 
able to rely on are the features in ring N, performance in ring N+1 isn't 
that important.

One of the several flaws in this model is what to do about the zero point. 
For example we might assume that everyone agrees that running ECMAScript 
should be a given. But it is quite possible to find serious, 
interoperability-affecting, incompatibilities in the way that browsers 
order scripts relative to each other. So we are building on shaky 
foundations.

> I also don't think that failing an inner ring should halt testing on 
> outer rings. The web developer should be able to see the level of 
> compatibility across several rings at once. For example, if a developer 
> is trying to build an app that needs a ring 997 feature (let's say 
> 'teleportation') they should be able to see if that works even if the 
> browser has a bug with vertical Hiragana text in ring 42.
>
> I also think we should grade the compatibility (not binary gray/green.)

This actually highlights an important issue. It is very difficult to work 
out a presentation of test results that will tell you if a browser is 
compatible-enough in its implementation of a given technology that you are 
likely to be able to use it without running into bugs. Certainly expecting 
a 100% pass rate is unreasonable; that typically just puts pressure on 
people to make crappy tests (this is something that you see with the "all 
tests must pass" conditions to transition specs to PR status). Moreover, 
it is generally possible to have a good level of interoperability with 
some level of bugginess at the edges. But simply taking a percentage score 
is no use either; one single test might be for a critical, frequently 
encountered case, so that 99% pass but a fail in that test meant a useless 
implementation but a 90% rate pass with all the failures in less important 
areas of the spec would be a better implementation.

Received on Sunday, 6 May 2012 11:46:39 UTC