Re: UAs passing tests if they don't implement a feature

Le Lun 25 juin 2012 19:15, Bjoern Hoehrmann a écrit :
> * Gérard Talbot wrote:
>>As a test writer, I want the test reviewer (or peers or anyone
>> examining
>>the source code) not to hesitate, not to search, not to guess, not to
>>think, not to calculate. I want to ease his/her tasks, efforts, to
>>optimize his/her time, etc.
>
> As a test writer, I want my tests to expose bugs or other problems. It
> should be up to "professionals" to ensure submitted tests meet all the
> stylistic criteria they care about. Precisely because it is easy to fix
> some test to meet all the style guidelines, it should not be up to sub-
> mitters to adhere to them, short of them doubling as test reviewer or
> maintainer or whatever, as learning all the latest the rules is hard.

I agree with you if a test submitter submits 5, 10, 20 tests.
I disagree with you if a particular test submitter submits 100 tests or
more. (S)he ought to know most if not all the guidelines and format
rules of submiting tests ... and this, of course, starting with the most
important ones. (S)he ought to get a review after submitting 10, 20
tests or so, so that no one has to retrofit his/her 21st (and up) tests
later.

I'll say this again, Bjoern: if you want to submit tests to the CSS test
suite, I can help and assist you so that your tests will comply with

Test format
http://wiki.csswg.org/test/format

Reviewing tests
http://wiki.csswg.org/test/review

CSS Test Review Checklist (more detailed review checklist)
http://wiki.csswg.org/test/css2.1/review-checklist


> You will not find, for instance, a browser vendor that tells bug repor-
> ters to re-submit sample code that exhibits some bug so the code con-
> forms to all the CSS WG style guidelines as of this week.

No, but...

All browser vendors have a "How to submit a good bug report" kind of
document to read, including a "How to create a reduced testcase"
webpage. And, at least at bugs.kde.org, a bug report which has no test
submitted or which has a test that does not trigger web standards
compliant rendering mode will get a low priority.

> There is no
> reason for the CSS WG's test suites to be run any other way, other than
> the lack of people who make a living fixing up tests to conform to the
> style guidelines.

I am not sure what you mean here.


> Take https://bugzilla.mozilla.org/show_bug.cgi?id=571708 as a simple
> example. That's a perfectly good bug report, you can read half a dozen
> of lines of code and see that they should have no effect on what the bug
> is about, `ctx.translate` and `ctx.fillRect` should not affect what the
> `ctx.fillStyle` property is set to, so setting `ctx.fillStyle` to the
> value that it has already been set to should have no effect. It does,
> though, very obviously so, so there is a bug. I should be able to submit
> this bug "to the W3C" without complaints how it's not a "reftest"

Well, the original test author is most likely the ideal person who can
best make reftest(s) for his own test(s).

> or how
> my variable naming conventions do not match some guidelines

This is mostly to help reviewers, examiners, others or even yourself
months, years later as you re-examine your own tests.

Variables names should help/contribute to the understanding of the code
and not be confusing or counter-intuitive, counter-meaningful.

> or about how
> I do not have <link> elements linking the sections in the relevant (non-
> existant) Recommendation or whatever else might be wrong with the bug.

Linking to sections of the spec is not a difficult task to do; I fail to
see why you seem to be annoyed with such things.

> Submitting it to Mozilla instead of the W3C is not a good use of my time
> as other browsers might have or might introduce the same bug, it would
> be much better to have it as part of the W3C test suite as then I could
> be reasonably sure that I will not encounter it again, or at least have
> a simple way to check whether I will, assuming that browser vendors will
> incorporate W3C Tests into their test suites and issue reports of some
> kind. Also note that Mozilla have not fixed the bug in two years, even
> though there is not much wrong with the bug report stylistically. There
> is no reason to expect the bug would be fixed by now if I had submitted
> a "better" bug report.

Maybe but, in some other cases, maybe not. Better bug report or better
test demonstrating the issue?


> When people find it trivial to come up with this kind of bug report, and
> when the CSS Working Group gets swamped with test case submissions,

We're already at 13699 tests now. And I don't know how many out of 13699
have been reviewed so far.

> it
> may be reasonable to impose more stringent quality policies, but as it
> is, there is more of a coverage problem, timely coverage in particular,
> so we do not suffer undue economic losses due to browser bugs.
>
> Test cases should be actionable. Concerns beyond that should be secon-
> dary, unimportant, so long as as there are far too few test cases that
> should be acted upon, except to root out un-actionable test cases.

There is a growing confusion with what are so-called stylistic
guidelines. Sometimes, it's really stylistic like how the test author
indent code; sometimes, it's not preferences like a tab width or how you
indent code but more serious issues (eg extraneous declarations) which
can create false positives, false negatives.

Gérard
-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 Test suite RC6, March 23rd 2011:
http://test.csswg.org/suites/css2.1/20110323/html4/toc.html

CSS 2.1 test suite harness:
http://test.csswg.org/harness/

Contributing to to CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html

Received on Tuesday, 26 June 2012 01:38:25 UTC