Re: UAs passing tests if they don't implement a feature

On Jun 20, 2012, at 12:33 AM, Aryeh Gregor wrote:

> On Tue, Jun 19, 2012 at 11:51 PM, Linss, Peter <peter.linss@hp.com> wrote:
>> The answer to this lies in the question, what are you testing for? If you're testing for _not_ implementing a feature, i.e. rotate with a percentage value, then an implementation that doesn't support transform should pass. If you're testing for rejecting percentages in an implementation of rotate, then a non-supporting implementation should not pass…
> 
> I'm not sure what this means.
> 
>> In this particular case, I second Simon's recommendation of a != ref to force a fail in UAs that don't implement transform at all. I'd think the != ref should be the rotated version (which should not be rendered via a rotate…). The two ways I can think to do that are: 1) render the rotated version in SVG using pre-computed coordinates, or 2) change your rotation to something that would be 90deg (25%?) and use a rectangular object as your rotation target.
> 
> I don't understand.  What would the purpose be of a != ref that tests
> that the object isn't rotated by 10% (whatever that means)?  In this
> hypothetical case, there's already an == ref of the object unrotated.
> Any UA that passes that will obviously pass the != ref, so what does
> it add?

What it adds is forcing a fail for the test in UAa that don't support transforms.

> 
> Or take a real example:
> http://test.csswg.org/shepherd/testcase/transform-background-006  That
> tests that if you transform the body and specify a background on the
> body, with no background on the root element, the background is not
> transformed (because the background is really rendered on the root
> element, which isn't transformed).  The reference is basically just
> the same file but without the transform.
> 
> What change would you suggest I make there?  Adding a != ref that has
> the background rotated wouldn't add anything, because anything that
> passed the == ref would pass the != ref too.  

Your assertion that a != reference isn't necessary when there is a == reference is not true.

There are many cases where both == and != references are necessary, as well as cases where multiple == references. The reason you need that is when it's possible for the reference to fail in the same way as the test does, which would create a false positive for the test.

> I could add some text or
> something and verify that that's rotated to fail UAs that don't
> support transforms at all, but that would add material to the test
> that's not related to what it's testing.

I'm going to correct my previous statement about the != ref. In the rotate % case I think the != ref _should_ be done via rotation, as should your != ref for the example you gave.

Concretely, for transform-background-006, there should be a != reference which is the same as the == reference except put the 'transform: rotate(90deg);' on the html element along with the background (and remove the transform-origin).

This would cause UAs that don't implement transforms to render the == and != references identically and therefore fail the test.

> 
> What's wrong with a UA that doesn't support transforms at all passing
> this particular test-case?

Because it sends the wrong message. If a UA can pass a significant portion of a test suite by _not_ implementing the features under test, that's a confusing signal (especially to authors). Ideally, a UA that does not implement transforms at all should fail 100% of the transforms test suite. Doesn't that make sense?

Received on Wednesday, 20 June 2012 17:27:17 UTC