Re: 28 proposals to improve testcase writing guidelines

Le Jeu 8 décembre 2011 18:01, Bjoern Hoehrmann a écrit :
> * Gérard Talbot wrote:
>>Finally, if an IT corporation or a dedicated test author (eg Ian
>>Hickson) intends to submit more than 100 testcases, then I think such
>> IT
>>corporation or test author should read all relevant documents
>> (including
>>the one on my 28 proposals) about creating good testcases. So far, only
>>3 entities fit such profile and have submitted over 500 tests in the
>> CSS
>>2.1 test suite: Microsoft, Mozilla and Ian Hickson.
>
> The goal of the test suite is to identify implementation differences of
> a certain kind, as many of them as possible, and as early as possible.
> The ideal test case shows that Firefox does one thing and Webkit another
> before Microsoft or Opera or whoever else adds a third implementation to
> the mix so the CSS Working Group can make it very clear which behavior,
> or behaviors, are correct, while it's cheap to make adjustments. What's
> needed beyond that is mainly something that allows you to get a rough
> idea about coverage. Anything else, filenames, specification links, you
> name it, results in fewer and later tests, as many browser vendors are
> not investing all that much in contributing to W3C's CSS test suites,
> and more requirements are not likely to make them invest more.

Quality of tests and quality of test making is going to impact quality,
reliablity and trustworthiness of results too. And quality testing
wraps, encompasses a lot of things, factors, issues and definitely not
just filenames and specification links.

Those browser vendors want reliable, trustworthy results too, you know.

Personally I can list tests which gave bad results, unreliable and
untrustworthy results because they miss "this", "that" or "x", "y" over
here and "z over there.
There are tests which are not testing what they are claiming to be testing.
There are tests which can not fail.
There are tests which are basic, simple; albeit coverage of simple tests
is, at some point, necessary, it nevertheless does not replace
realistical webpage coding scenario.

There are many tests which use extraneous elements and unneeded CSS
declarations and, as such, - sometimes - they can have unexpected and
unforeseen side effects on the goal/task/target that the test was trying
to achieve/execute/reach.

And despite all of the above, there is not a lot of tests which, I
believe, can not be rehabilitated, could not be corrected, improved.


> Have a look at <http://shadowregistry.org/js/misc/>.

[snipped]

> If I could submit tests
> for CSS features like that, without all the overhead of learning how to
> name test files and whatever, I would have submitted a lot.


Bjoern, submit your CSS testcases! Put them in a webpage and then give
an URL about these in this mailing list. (I assume here that they are
CSS 2.1 tests) And forget about how to name such test files. I have not
found a single test which had a truly bad - detrimentally speaking -
filename.


> All I would
> have to do to add a new test case to the system above is appending a few
> lines of code to <http://shadowregistry.org/js/misc/tests.txt> and that
> is already rather inconvenient as I would have to come up with a "title"
> for the new test case.


If you can come up with good testcases, worthy testcases, useful
testcases, then I assure that finding good filenames and good titles for
these tests (and fixing a few other issues which they may have) is going
to be a very easy task.

"
if someone has only a few tests to submit, it's probably better for
Gérard to shepherd them; that way they don't get lost or give up halfway
through the review process.
"
January 7th 2010
http://www.css3.info/css-quality-assurance/#comment-257679


"If your testcase does not comply with such guidelines or such format, I
will help and assist you."
January 7th 2010
http://web.archive.org/web/20100113232548/http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html


regards, Gérard
-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 Test suite RC6, March 23rd 2011:
http://test.csswg.org/suites/css2.1/20110323/html4/toc.html

CSS 2.1 test suite harness:
http://test.csswg.org/harness/

Contributing to to CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html

Received on Friday, 9 December 2011 04:31:01 UTC