Re: Feature Rings, Incomplete Rings, and Compliance Grading

I'm suggesting that the current system is pretty easy to game, an example from the CSS text tests:

  assert( H.test.cssProp( elem, "wordBreak" ), "wordBreak supported" );

We can "fix" Firefox to pass that test and go 100% green on ring 0 without having any word breaking at all. The expensive part of wordBreak isn't exposing the "wordBreak" string in the JS API's (yet that's all you need to pass!) It's premature to assert that you're testing for wordBreak with the tests as coded. I would leave it off any rings until it actually does something. 

That's my concern about the rings as they are pass/fail. In my example, the expensive part of getting a "100% correct" wordBreak implementation is probably breaking words with complex ligatures on right-to-left Arabic text. Will ringmark really "fail" a browser with a zero score for wordBreak on such a test?  Let's give them grades instead.

-- Jet


----- Original Message -----
From: "Tobie Langel" <tobie@fb.com>
To: "Jet Villegas" <jet@mozilla.com>
Cc: robin@berjon.com, public-coremob@w3.org, jason@cloudfour.com, "jeanfrancois moy" <jeanfrancois.moy@orange.com>, "Charles McCathieNevile" <chaals@opera.com>
Sent: Thursday, May 3, 2012 4:24:25 PM
Subject: Re: Feature Rings, Incomplete Rings, and Compliance Grading

On 5/4/12 12:27 AM, "Jet Villegas" <jet@mozilla.com> wrote:

>I think there is great value in promoting a mobile-centric feature-driven
>matrix. Not all mobile phone web apps require all of ring 0, but a
>specific app may well need a feature in an outer ring or completely fail.
>Authors should be able to see how much of ring x is available even as
>lower rings aren't filled out. In other words, let all the tests run.

That's easy, just navigate to: http://rng.io/?all


>As far as features go, it seems that the current ring organization
>encourages vendors to get "green" and not actual spec compliance.

I don't understand how these two aspects are related. If you're suggesting
the tests aren't complete, I fully agree and your contribution are welcome.

>I'd like to see real grading there and not a boolean "pass" condition.
>This will make correctness of a feature weigh more than its presence.

I'm not sure what you mean by this either, unfortunately.

--tobie

Received on Sunday, 6 May 2012 05:25:13 UTC