W3C home > Mailing lists > Public > public-css-testsuite@w3.org > April 2014

Some proposals on Test Review Checklist and Test Style Guidelines

From: Gérard Talbot <css21testsuite@gtalbot.org>
Date: Tue, 15 Apr 2014 06:56:22 -0400
To: Public CSS Test suite mailing list <public-css-testsuite@w3.org>, Rebecca Hauck <rhauck@adobe.com>
Message-ID: <1e994ef41923a4cd358cc0d13653174f@gtalbot.org>


Test Review Checklist

I believe this page may be source of confusion.

"All tests" is supposed to mean a) non-self-describing tests and b) 
self-describing tests;  each ( a and b ) categories of tests could be of 
type manual (not a reftest) or automatable (a reftest).

When you refer to "Reftests", I believe you mean tests that have an(or 
several) associated reference file(s). But this may not be what people 
would think...

Here's my proposal:


All tests

The test passes when it's supposed to pass.

The test fails when it's supposed to fail.

The test is testing what it thinks it's testing.

The spec backs up the expected behavior in the test.

The test is automated as either reftest or a script test unless there's 
a very good reason why the test must be manual.

The test does not use external resources.

The test does not use proprietary features (vendor-prefixed or 

The title is descriptive but not too wordy.

The test is as cross-platform as reasonably possible, working across 
different devices, screen resolutions, paper sizes, etc.

Self-describing tests
The self-describing statement is clear, short and self-explanatory. Your 
mother/husband/roommate/brother/bus driver should be able to say whether 
the test passed or failed within a few seconds, and not need to spend 
several minutes thinking or asking questions.

Reference file only
The reference file is accurate and will render pixel-perfect identically 
to the test on all platforms.

The reference file uses a different technique that won't fail in the 
same way as the test.

Script Tests Only
no change; same as now

In depth Checklist
no change; same as now


Note that I am also proposing some changes.
"The self-describing statement is accurate, precise, simple, and 
is now
"The self-describing statement is clear, short and self-explanatory."

Also, I propose to remove
"If there are limitations (e.g. the test will only work on 96dpi 
devices, or screens wider than 200 pixels), then these are documented in 
the instructions."


If a test is supposed to only work in a 96dpi device or paper media only 
or etc.., then test creators only need to use accordingly this list


This example
coming from
is not best (and that's my fault!) because green is used without red in 
case of a failure. I would need to change those 
border-bottom-applies-to-* tests so that they would use
border-bottom-width-applies-to-* tests. At the same time, they would 
reuse the same reference files.


Test Style Guidelines

"This line should ..."
I am for replacing "line" by "sentence" or by "text" and to keep "line" 
for linear test situations.
I am also strongly for systematic usage of the "Test passes if ..." 
introductory words in every examples of self-describing sentences.

I am for replacing
"Test passes if there is a green square and no red."
"Test passes if there is a filled green square and *no red*."
is already referenced by 149 tests and there is no reason why it would 
not or could not be referenced by thousands of tests.

" on this page"
in self-describing sentences can be safely removed. Same thing with
"you can see"
"You should see"
"you can view"
"in this page"
"on this page"
"below this line"
"after this line"
"below this sentence"
"after this sentence"
"below this paragraph"
"under this paragraph"
"in the next paragraph"
"after this"
"which follows"

If self-describing tests all start with the recommended "Test passes if 
..." and if testers are assumed to not be blind, then all these 
expressions can safely be removed.

d) "Filler text" should be preferred for page background; "Text sample" 
should be preferred when a text is being the object of the test.

Web authors' contributions to CSS 2.1 test suite
CSS 2.1 Test suite RC6, March 23rd 2011
Received on Tuesday, 15 April 2014 10:57:00 UTC

This archive was generated by hypermail 2.4.0 : Friday, 20 January 2023 19:58:20 UTC