Re: UAs passing tests if they don't implement a feature

On Wed, Jun 20, 2012 at 8:26 PM, Linss, Peter <peter.linss@hp.com> wrote:
> There are many cases where both == and != references are necessary, as well as cases where multiple == references. The reason you need that is when it's possible for the reference to fail in the same way as the test does, which would create a false positive for the test.

That's true, and in those cases I agree a != ref is appropriate.  But
in cases where a UA renders both the test and reference *correctly*,
despite not implementing the feature, I think there's no problem with
it passing.

> I'm going to correct my previous statement about the != ref. In the rotate % case I think the != ref _should_ be done via rotation, as should your != ref for the example you gave.
>
> Concretely, for transform-background-006, there should be a != reference which is the same as the == reference except put the 'transform: rotate(90deg);' on the html element along with the background (and remove the transform-origin).
>
> This would cause UAs that don't implement transforms to render the == and != references identically and therefore fail the test.

That's possible, sure.  Really, though, the != ref in this case is red
herring -- it doesn't have anything to do with what's being tested, if
you think about it.  I could also add a != ref with 'transform:
scale(3.5) translateX(-21em)', and it would have exactly the same
effect.  All that tests is that transforms do something.  (In general,
all notrefs test are "this change does *something*", without testing
what.)

> Because it sends the wrong message. If a UA can pass a significant portion of a test suite by _not_ implementing the features under test, that's a confusing signal (especially to authors). Ideally, a UA that does not implement transforms at all should fail 100% of the transforms test suite. Doesn't that make sense?

I see what you're saying from a marketing perspective, but not a
technical one.  On a technical level, requirements of the form "ignore
the transform in case X" are actually implemented correctly by a UA
that doesn't implement transforms at all, and I think it's fine for
the test suite to reflect that.  Particularly if it requires modifying
tests to incorporate requirements unrelated to what's being directly
tested.

If we're worried about author perception, well, every UA that authors
care about already supports transforms (albeit perhaps prefixed), so I
really think that's moot here.

At the end of the day, if people want to add extra != refs to the
handful of my tests that would otherwise pass in non-supporting UAs,
they can go ahead.  But once again, this is something where reasonable
people can differ -- and it seems there are multiple members of this
mailing list who hold each position -- so I really don't think it's
valid grounds for not approving a test.

This is the kind of thing that strongly discourages contributions to
the CSS testsuite: requiring test submitters to repeatedly rewrite
submitted tests that are technically correct and serve their purpose
perfectly well, based on long lists of written and unwritten rules
that are not disclosed in advance of the test submissions and that the
submitter may or may not think make sense.  That kind of policy works
when submitters are forced to work with the CSSWG, but not otherwise.

But for now, I have tests to submit.  And adding some != refs here is
a lot less onerous than rewriting all my style attributes as <style>
tags, which I'm also spending hours doing because the CSSWG apparently
feels my time is better spent doing that than actually writing
substantive new tests (or other useful work).  So I guess it's easier
to just do it instead of arguing about it, given that I've already put
all the work into writing the tests to start with and they're not very
useful if they're not accepted.  This is not the kind of policy that
is likely to prompt me to write any new CSS tests in the future,
however.

Received on Thursday, 21 June 2012 12:54:14 UTC