Re: UAs passing tests if they don't implement a feature

On 06/19/2012 08:36 AM, Aryeh Gregor wrote:
> A bunch of my submitted transforms tests tested something like
> "'transform: rotate(45deg)' is the same as 'transform:
> rotate(0.125turn)'", where the most logical way to construct the test
> was to have the test and reference file be identical except that they
> used a different transform rule.  Simon Fraser pointed out on Shepherd
> (separately for a lot of different tests) that this meant a UA that
> didn't implement transforms at all would pass the test, because it
> wouldn't apply the transform to either.  Likewise if it just didn't
> implement rotate(), or didn't implement the deg or turn units, etc.
> He suggested adding mismatch refs to verify that the transform wasn't
> being simply ignored.
>
> I just want to check with the broader group to see if this is a
> guideline we want.  If so, we should add it to the test guidelines.
> It's not necessary if the purpose of tests is to alert implementers to
> possible bugs in their implementations -- this is how I was viewing
> it.  Implementers who don't implement transforms at all won't even
> bother running the tests.
>
> On the other hand, if the purpose of the tests is at least partly so
> that users can compare standards conformance of different
> implementations, then we do want UAs that don't implement transforms
> at all to fail all these tests.  Otherwise, a partial implementation
> might cause the score on the tests to go *down* relative to no
> implementation, which from this perspective would be bad.

I think where it's straightforward to have implementations that
don't support the feature to fail, we should make them fail.
Not refs make this relatively easy for many tests.

If handling that convolutes the test significantly, though,
I don't see this as a requirement.

> Relatedly, I would like people's opinions on the following: I do think
> that either way, it's fine for a test to pass in a UA that doesn't
> implement the feature if a conforming UA is supposed to do nothing
> too.  E.g., if I have a test that "transform: rotate(10%)" is a parse
> error, then a non-conforming UA *should* pass that test -- the
> property is supposed to not be supported in this case.  I wanted to
> bring this up too to see what everyone else thinks, because Simon
> suggested adding != refs in these cases too.  (It's not clear to me
> what the != ref would even be.)

If you're checking for parse errors, you want to check not only
that the erroneous declaration has no effect, but also that it
is thrown out and therefore doesn't override any previous
declarations. So it would actually be better to have this test's
output depend on a transform applying. At which point you can
easily add a not ref.

On the other hand, your example with the background on body
propagating to the root and therefore not being transformed...
I don't see a problem with allowing that to pass in implementations
that don't support Transforms. They are correctly propagating the
body background to the root, and that's fundamentally what you're
testing anyway.

~fantasai

Received on Wednesday, 20 June 2012 18:03:32 UTC