Re: UAs passing tests if they don't implement a feature

Le Lun 25 juin 2012 19:15, Bjoern Hoehrmann a écrit :

> As a test writer, I want my tests to expose bugs or other problems. It
> should be up to "professionals" to ensure submitted tests meet all the
> stylistic criteria they care about. Precisely because it is easy to fix
> some test to meet all the style guidelines, it should not be up to sub-
> mitters to adhere to them, short of them doubling as test reviewer or
> maintainer or whatever, as learning all the latest the rules is hard.
>
> You will not find, for instance, a browser vendor that tells bug repor-
> ters to re-submit sample code that exhibits some bug so the code con-
> forms to all the CSS WG style guidelines as of this week. There is no
> reason for the CSS WG's test suites to be run any other way, other than
> the lack of people who make a living fixing up tests to conform to the
> style guidelines.

[snipped]

Over the years, browser vendors, in their respective bug tracking
systems, have introduced many keywords to compensate poor quality bug
reports: needsreduction, hasreduction, needsinfo, waitingforinfo, etc.

and new status field: INCOMPLETE

and new resolution field: EXPIRED

and they also have elaborated pre-edited "canned" responses and also
auto/robo-resolving plans and strategies to compensate for hundreds and
thousands of unactionable or unconfirmable bug reports piling up from
bug reporters during years and who do not follow bug writing guidelines,
help etiquette, etc..

I know these issues pretty well as I have reported bug reports to
Mozilla, Microsoft, Opera, Safari, KDE and done QA triaging for Mozilla
and KDE in the last 12 years.

After a while, volunteer+benevolent QA triaging people really get tired
too of reading+dealing with bug reports that:
- have poor description, general summary
- have vague, abstract data, absence or lack of specifics
- have very often no testcase because creating a testcase is often
impossible and here, I’m not even speaking of a reduced or minimal
testcase
- sometimes, no url provided
- if an url is provided, then you get to see
   o- a very long web page (over 50KB),
   o- with insane amounts of nested tables,
   o- tag soup,
   o- invalid markup/CSS code,
   o- IE-specific code,
   o- no doctype declaration
   o- trigger backward-compatible "quirks" rendering mode or one of the
numerous specific IE compat modes
   o- hair-rising amount of javascript code,
   o- undecipherable jQuery, dojo, mootools kinds of javascript libraries
   o- scripts which were copied and pasted (so even the "webmaster"
can’t even understand what he stole here and there)
etc..

I'm convinced that browser vendors are looking for/prefer good bug
reports with good tests.

It is in the best interests of bug reporters and of test authors
(submitting into W3C CSS test suites or to browser vendors) to comply as
much as they can with stated expectations, written guidelines, defined
format rules in related documents and to do their fair/reasonable share
of efforts in that sense.

> when the CSS Working Group gets swamped with test case submissions,
> it may be reasonable to impose more stringent quality policies

Isn't the CSS Working Group already swamped with test case submissions?
Aren't we at that point already?
"There are currently 13701 testing assets under management for 15
specifications."
http://test.csswg.org/shepherd/

Peter Linss recently said that there are only 3 test reviewers in CSSWG...

Anyway, why wait to get "swamped" with test case submissions before
defining format, guidelines, rules, review process, etc?

Gérard
-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 Test suite RC6, March 23rd 2011:
http://test.csswg.org/suites/css2.1/20110323/html4/toc.html

CSS 2.1 test suite harness:
http://test.csswg.org/harness/

Contributing to to CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html

Received on Tuesday, 26 June 2012 17:48:57 UTC