Re: UAs passing tests if they don't implement a feature

* Gérard Talbot wrote:
>As a test writer, I want the test reviewer (or peers or anyone examining
>the source code) not to hesitate, not to search, not to guess, not to
>think, not to calculate. I want to ease his/her tasks, efforts, to
>optimize his/her time, etc.

As a test writer, I want my tests to expose bugs or other problems. It
should be up to "professionals" to ensure submitted tests meet all the
stylistic criteria they care about. Precisely because it is easy to fix
some test to meet all the style guidelines, it should not be up to sub-
mitters to adhere to them, short of them doubling as test reviewer or
maintainer or whatever, as learning all the latest the rules is hard.

You will not find, for instance, a browser vendor that tells bug repor-
ters to re-submit sample code that exhibits some bug so the code con-
forms to all the CSS WG style guidelines as of this week. There is no
reason for the CSS WG's test suites to be run any other way, other than
the lack of people who make a living fixing up tests to conform to the
style guidelines.

Take https://bugzilla.mozilla.org/show_bug.cgi?id=571708 as a simple
example. That's a perfectly good bug report, you can read half a dozen
of lines of code and see that they should have no effect on what the bug
is about, `ctx.translate` and `ctx.fillRect` should not affect what the
`ctx.fillStyle` property is set to, so setting `ctx.fillStyle` to the
value that it has already been set to should have no effect. It does,
though, very obviously so, so there is a bug. I should be able to submit
this bug "to the W3C" without complaints how it's not a "reftest" or how
my variable naming conventions do not match some guidelines or about how
I do not have <link> elements linking the sections in the relevant (non-
existant) Recommendation or whatever else might be wrong with the bug.

Submitting it to Mozilla instead of the W3C is not a good use of my time
as other browsers might have or might introduce the same bug, it would
be much better to have it as part of the W3C test suite as then I could
be reasonably sure that I will not encounter it again, or at least have
a simple way to check whether I will, assuming that browser vendors will
incorporate W3C Tests into their test suites and issue reports of some
kind. Also note that Mozilla have not fixed the bug in two years, even
though there is not much wrong with the bug report stylistically. There
is no reason to expect the bug would be fixed by now if I had submitted
a "better" bug report.

When people find it trivial to come up with this kind of bug report, and
when the CSS Working Group gets swamped with test case submissions, it
may be reasonable to impose more stringent quality policies, but as it
is, there is more of a coverage problem, timely coverage in particular,
so we do not suffer undue economic losses due to browser bugs.

Test cases should be actionable. Concerns beyond that should be secon-
dary, unimportant, so long as as there are far too few test cases that
should be acted upon, except to root out un-actionable test cases.
-- 
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

Received on Monday, 25 June 2012 23:15:53 UTC