Re: UAs passing tests if they don't implement a feature

On Jun 19, 2012, at 8:36 AM, Aryeh Gregor wrote:

> A bunch of my submitted transforms tests tested something like
> "'transform: rotate(45deg)' is the same as 'transform:
> rotate(0.125turn)'", where the most logical way to construct the test
> was to have the test and reference file be identical except that they
> used a different transform rule.  Simon Fraser pointed out on Shepherd
> (separately for a lot of different tests) that this meant a UA that
> didn't implement transforms at all would pass the test, because it
> wouldn't apply the transform to either.  Likewise if it just didn't
> implement rotate(), or didn't implement the deg or turn units, etc.
> He suggested adding mismatch refs to verify that the transform wasn't
> being simply ignored.
> 
> I just want to check with the broader group to see if this is a
> guideline we want.  If so, we should add it to the test guidelines.
> It's not necessary if the purpose of tests is to alert implementers to
> possible bugs in their implementations -- this is how I was viewing
> it.  Implementers who don't implement transforms at all won't even
> bother running the tests.
> 
> On the other hand, if the purpose of the tests is at least partly so
> that users can compare standards conformance of different
> implementations, then we do want UAs that don't implement transforms
> at all to fail all these tests.  Otherwise, a partial implementation
> might cause the score on the tests to go *down* relative to no
> implementation, which from this perspective would be bad.

If a test is testing for the proper implementation of a feature, then it shouldn't be possible for an implementation that doesn't support the feature at all to pass.

> 
> 
> Relatedly, I would like people's opinions on the following: I do think
> that either way, it's fine for a test to pass in a UA that doesn't
> implement the feature if a conforming UA is supposed to do nothing
> too.  E.g., if I have a test that "transform: rotate(10%)" is a parse
> error, then a non-conforming UA *should* pass that test -- the
> property is supposed to not be supported in this case.  I wanted to
> bring this up too to see what everyone else thinks, because Simon
> suggested adding != refs in these cases too.  (It's not clear to me
> what the != ref would even be.)

The answer to this lies in the question, what are you testing for? If you're testing for _not_ implementing a feature, i.e. rotate with a percentage value, then an implementation that doesn't support transform should pass. If you're testing for rejecting percentages in an implementation of rotate, then a non-supporting implementation should not pass…

In this particular case, I second Simon's recommendation of a != ref to force a fail in UAs that don't implement transform at all. I'd think the != ref should be the rotated version (which should not be rendered via a rotate…). The two ways I can think to do that are: 1) render the rotated version in SVG using pre-computed coordinates, or 2) change your rotation to something that would be 90deg (25%?) and use a rectangular object as your rotation target.

Peter

Received on Tuesday, 19 June 2012 20:53:18 UTC