Re: UAs passing tests if they don't implement a feature

Hi Aryeh,

On Jun 19, 2012, at 8:36 AM, Aryeh Gregor wrote:

> A bunch of my submitted transforms tests tested something like
> "'transform: rotate(45deg)' is the same as 'transform:
> rotate(0.125turn)'", where the most logical way to construct the test
> was to have the test and reference file be identical except that they
> used a different transform rule.  Simon Fraser pointed out on Shepherd
> (separately for a lot of different tests) that this meant a UA that
> didn't implement transforms at all would pass the test, because it
> wouldn't apply the transform to either.  Likewise if it just didn't
> implement rotate(), or didn't implement the deg or turn units, etc.
The current style guidelines disallow these kind of tests, just because of that reason.

> He suggested adding mismatch refs to verify that the transform wasn't
> being simply ignored.
As long as you transform something, that is definitely a great idea. This could limit the restriction for these kind of tests. Just if all ==ref and !=ref pass, the whole test would pass.

> 
> I just want to check with the broader group to see if this is a
> guideline we want.  If so, we should add it to the test guidelines.
> It's not necessary if the purpose of tests is to alert implementers to
> possible bugs in their implementations -- this is how I was viewing
> it.  Implementers who don't implement transforms at all won't even
> bother running the tests.
What if just a sub feature is not supported, like applying transform property on SVG elements. This is not supported by any implementation, even if transforms is widely supported.

> 
> On the other hand, if the purpose of the tests is at least partly so
> that users can compare standards conformance of different
> implementations, then we do want UAs that don't implement transforms
> at all to fail all these tests.  
Agree.

> Otherwise, a partial implementation
> might cause the score on the tests to go *down* relative to no
> implementation, which from this perspective would be bad.
> 
> 
> Relatedly, I would like people's opinions on the following: I do think
> that either way, it's fine for a test to pass in a UA that doesn't
> implement the feature if a conforming UA is supposed to do nothing
> too.  E.g., if I have a test that "transform: rotate(10%)" is a parse
> error, then a non-conforming UA *should* pass that test -- the
> property is supposed to not be supported in this case.  I wanted to
> bring this up too to see what everyone else thinks, because Simon
> suggested adding != refs in these cases too.  (It's not clear to me
> what the != ref would even be.)
> 
This is an interesting question. The negative ref test won't help here. So it wouldn't matter how many refs we add, an implementation would always pass. On the other side, this is the correct behavior. The rotation is not applied. If implementations support transforms or not, the element should not be transformed. But for these kind of tests the ref doesn't need to apply a transform at all, does it? The exception would be the negative rotate. A ref file that you can easily work around. We did it for SVG transforms already.

Greetings,
Dirk

Received on Tuesday, 19 June 2012 16:03:41 UTC