- From: James Graham <jgraham@opera.com>
- Date: Sun, 6 May 2012 13:46:04 +0200 (CEST)
- To: Jet Villegas <jet@mozilla.com>
- cc: public-coremob@w3.org
On Wed, 2 May 2012, Jet Villegas wrote: > I would prefer that the rings represent features like ring 0 == text, > ring 1 == images, ring 2 == JavaScript, ring 3 == audio, ring 4 == > video... So I think that makes a lot of sense for a testsuite. However I don't think the primary goal of this group is to produce a testsuite, although I certianly encourage the production of tests as part of the work. It seems to me that this group exists to set an agenda; that is to get the relevant stakeholders from the worlds of content production, browser development and web standards together to put together a rough priority order on nascent web technologies so that everyone is pulling in the same direction; browser vendors are implementing the things that authors want the most, standards committees are putting the most effort into stabalising the things that are being implemeneted, and authors are desigining sites using cross-browser technologies rather than a smorgasboard of UA-specific hacks. Given this outlook; the ringed approach makes a great deal more sense because at long as we agree that the next thing that we want authors to be able to rely on are the features in ring N, performance in ring N+1 isn't that important. One of the several flaws in this model is what to do about the zero point. For example we might assume that everyone agrees that running ECMAScript should be a given. But it is quite possible to find serious, interoperability-affecting, incompatibilities in the way that browsers order scripts relative to each other. So we are building on shaky foundations. > I also don't think that failing an inner ring should halt testing on > outer rings. The web developer should be able to see the level of > compatibility across several rings at once. For example, if a developer > is trying to build an app that needs a ring 997 feature (let's say > 'teleportation') they should be able to see if that works even if the > browser has a bug with vertical Hiragana text in ring 42. > > I also think we should grade the compatibility (not binary gray/green.) This actually highlights an important issue. It is very difficult to work out a presentation of test results that will tell you if a browser is compatible-enough in its implementation of a given technology that you are likely to be able to use it without running into bugs. Certainly expecting a 100% pass rate is unreasonable; that typically just puts pressure on people to make crappy tests (this is something that you see with the "all tests must pass" conditions to transition specs to PR status). Moreover, it is generally possible to have a good level of interoperability with some level of bugginess at the edges. But simply taking a percentage score is no use either; one single test might be for a critical, frequently encountered case, so that 99% pass but a fail in that test meant a useless implementation but a 90% rate pass with all the failures in less important areas of the spec would be a better implementation.
Received on Sunday, 6 May 2012 11:46:39 UTC