UAs passing tests if they don't implement a feature

A bunch of my submitted transforms tests tested something like
"'transform: rotate(45deg)' is the same as 'transform:
rotate(0.125turn)'", where the most logical way to construct the test
was to have the test and reference file be identical except that they
used a different transform rule.  Simon Fraser pointed out on Shepherd
(separately for a lot of different tests) that this meant a UA that
didn't implement transforms at all would pass the test, because it
wouldn't apply the transform to either.  Likewise if it just didn't
implement rotate(), or didn't implement the deg or turn units, etc.
He suggested adding mismatch refs to verify that the transform wasn't
being simply ignored.

I just want to check with the broader group to see if this is a
guideline we want.  If so, we should add it to the test guidelines.
It's not necessary if the purpose of tests is to alert implementers to
possible bugs in their implementations -- this is how I was viewing
it.  Implementers who don't implement transforms at all won't even
bother running the tests.

On the other hand, if the purpose of the tests is at least partly so
that users can compare standards conformance of different
implementations, then we do want UAs that don't implement transforms
at all to fail all these tests.  Otherwise, a partial implementation
might cause the score on the tests to go *down* relative to no
implementation, which from this perspective would be bad.


Relatedly, I would like people's opinions on the following: I do think
that either way, it's fine for a test to pass in a UA that doesn't
implement the feature if a conforming UA is supposed to do nothing
too.  E.g., if I have a test that "transform: rotate(10%)" is a parse
error, then a non-conforming UA *should* pass that test -- the
property is supposed to not be supported in this case.  I wanted to
bring this up too to see what everyone else thinks, because Simon
suggested adding != refs in these cases too.  (It's not clear to me
what the != ref would even be.)

Received on Tuesday, 19 June 2012 15:37:38 UTC