Re: UAs passing tests if they don't implement a feature

On Jun 21, 2012, at 5:53 AM, Aryeh Gregor <ayg@aryeh.name> wrote:

> On Wed, Jun 20, 2012 at 8:26 PM, Linss, Peter <peter.linss@hp.com> wrote:
>> There are many cases where both == and != references are necessary, as well as cases where multiple == references. The reason you need that is when it's possible for the reference to fail in the same way as the test does, which would create a false positive for the test.
> 
> That's true, and in those cases I agree a != ref is appropriate.  But
> in cases where a UA renders both the test and reference *correctly*,
> despite not implementing the feature, I think there's no problem with
> it passing.

We talked about this at the TestTheWebForward meeting, which is where I was reviewing tests (shame you couldn't be there!).

You could imagine a test suite in which there is a single test that check whether the UA supports the feature, and every other test is a ref test, where the test and the ref render identically when the UA does not support the feature.

In this scenario, a UA has to pass ALL the tests before any statement can be made about the level of support; it's all or nothing. A partial pass means nothing. It also means that individual test passes can't be evaluated in isolation, which lessens the utility of the hooks into the spec that indicate test coverage per section etc. It also means that people doing reviewing in Shepherd were marking as passing in WebKit tests that used unprefixed transform, which was very misleading.

That's why I suggested adding != refs for the ref.

Simon

Received on Thursday, 21 June 2012 21:36:49 UTC