W3C home > Mailing lists > Public > public-css-testsuite@w3.org > June 2018

39 writing guidelines to improve CSS tests

From: Gérard Talbot <css21testsuite@gtalbot.org>
Date: Mon, 11 Jun 2018 18:26:55 -0400
To: Public CSS Test suite mailing list <public-css-testsuite@w3.org>
Message-ID: <49dbb0001d6ef25c5c27c3e42a631919@gtalbot.org>
Hello all,

Since 2010, I have examined, checked and reviewed hundreds of CSS tests. 
I wrote down all kinds of notes about what I found. The following 
guidelines are my recommendations. They are listed in no particular 
order. They are not some kind of biblical 39 commandements or anything 
that everyone must follow. Heck, some items could be debattable or minor 
or a matter of taste or plain wrong. And some of my own tests might even 
be in conflict with my own guidelines!

This is a follow-up/continuation of
28 proposals to improve testcase writing guidelines
http://lists.w3.org/Archives/Public/public-css-testsuite/2011Dec/0004.html

Overall, I am open to feedback, criticisms and any comments you may have 
on this document.


-  -  -  -  -  -  -  -  -


1-
Adequate font-size for :first-letter testcases
**********************************************

:first-letter testcases should all be using a much bigger font-size: 
say, at least 32px
Minimum 2em; recommended: 3em;

eg.
http://test.csswg.org/suites/css2.1/20110323/html4/first-letter-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/first-letter-dynamic-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/first-letter-dynamic-003a.htm
http://test.csswg.org/suites/css2.1/20110323/html4/first-letter-dynamic-003b.htm
http://test.csswg.org/suites/css2.1/20110323/html4/first-letter-nested-001.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/first-letter-quote-001.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/first-letter-quote-005.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/first-letter-selector-018.htm

Better
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/first-letter-001.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/first-letter-nested-001.htm


2-
font units and line-height
**************************

3pt == 4px; so best is to use a factor of 3 for pt unit: ideal is
12pt (or 15pt or 18pt, etc) for parent font size.

5ex (ahem) = 4em; so best is to use a factor of 5 for ex unit;
ideal is 20px (or 30px or 40px, etc) for parent font size.
The minimal font-size for all testcases involving ex unit and Ahem font 
should be 20px; and then a factor of 10: 20px, 30px, 40px, etc.

The ideal is to use a value which is going to be convertible to pixel
unit into whole integer, without creating fraction.

font: 20px/1.5 serif;
will initially be resorting to a font-size of 20px and a line-height of 
30px
For optimal convertability, even values are preferred because creating 
more
predictable layout.

Leading outside of content area should be dividable by 2 without a 
remainer.

font: 25px/50px Ahem;
is going to create problems as 25px is not dividable by 2 without a 
remainer.

Good is:
font: 25px/51px Ahem;
because 26px is dividable by 2 without a remainer.


3-
Adequate margin-left for list-marker testcases
**********************************************

Horizontal space should be made available in testcases to view bullet
list-marker, in particular in 'list-style-position: outside' testcases.
This should be part of pass condition of all 'list-style' testcases.
Minimum 1em; recommended: 2em.
Examples:
http://test.csswg.org/suites/css2.1/20110323/html4/after-content-display-003.htm
http://test.csswg.org/suites/css2.1/20110323/html4/universal-selector-005.htm


4-
Outline borders fully viewable
******************************

Parts of the outline can be clipped on the left and on the right.
The full width of outline should be viewable, visible in testcases, 
furthermore
if tests require to examine them or even compare them with a reference 
border.
Recommendation: when outline-width is, say, 20px, then set all 4 margins 
to 20px too.

E.g.:

http://test.csswg.org/suites/css2.1/20110323/html4/outline-001.htm

         <style type="text/css">
             #div1
             {
                 outline: green;
                 outline-style: dotted;
                 outline-width: 15px;
             }
         </style>
     </head>
     <body>
         <p>Test passes if there is a dotted outline that goes around the 
following 'Sample Text' and the "Filler Text" below is overlapped by the 
outline.</p>
         <div id="div1">Sample Text</div>
         <div>Filler Text</div>

would become

         <style type="text/css">
         div {margin: 15px;}

         #div1
             {
                 outline: green;
                 outline-style: dotted;
                 outline-width: 15px;
             }
         </style>


5-
"Filler text" versus "Text sample"
**********************************

"Filler text" should be for background.
(good) 
http://test.csswg.org/suites/css2.1/20110323/html4/background-bg-pos-207.htm

"Text sample" or "Text example" should be for testing.
(good) 
http://test.csswg.org/suites/css2.1/20110323/html4/content-type-000.htm

(bad)
<div>Filler text</div>
<div>Filler text</div>
<div>Filler text</div>

(good)
<div>1st text sample</div>
<div>2nd text sample</div>
<div>3rd text sample</div>

(good)
<div>Text example 1</div>
<div>Text example 2</div>
<div>Text example 3</div>


6-
Differentiable text content when using Ahem font
************************************************

When editing a text node which is going to be using Ahem font, then
give preference to
1234567890
instead of/rather than
XXXXXXXXXX
, especially if counting the characters is important in the testcase; 
that way,
test reviewer, test author when revisiting testcase much later, web 
authors do
not have to count anything. The string "1234567890" can also be much 
easier to
find or to test with web authoring debugging tools like IE Developer 
Tools,
Firefox's DOM inspector, Chrome Web Inspector, Safari Web Inspector, 
Opera
DragonFly, Konqueror's DOM treeviewer.

Another useful manner to edit text nodes is to give text content an 
useful,
meaningful, descriptive value in relationship with the purpose of the 
test: eg.
<span>MustBeAbove</span>
is going to be much more helpful, useful and appreciated than
<span>XXXXXXXXXXX</span>. Because it contributes to self-explanatory 
sense
of testcases while editing differentiable text content.

Use differentiable, unique text content whenever possible and wherever 
suitable so that examining the source code can be easier, so that the 
efforts of
understanding the concepts - sometimes difficult ones - and the testcase
itself (and its complexity) can be less demanding

E.g.:

(undifferentiated therefore less recommendable) <p>x <span>xxxxx 
xxxxx</span> x</p>

versus

(differentiated therefore more recommendable) <p>a <span>ceikl 
mnors</span> u</p>

E.g.:

   <div>shuldboverlaPPed<br>
   <span>bleedover1stline</span></div>
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/border-padding-bleed-001.htm

If a green rectangle indicating a passed test must have a text content, 
then
using "PASS" is more logical and useful than "X". If a green rectangle
must overlap a red rectangle with a text content, then using "FAIL" is 
more
logical and useful than "X". Again, this contributes to make source text 
content self-explanatory.


7-
Correct usage of 'border-collapse: collapse'
********************************************

It is okay to declare 'border-collapse: collapse' when table and/or 
sub-table components have borders which require, by the testcase, to be 
collapsed. It is *not* okay to use 'border-collapse: collapse' in order 
to neutralize or to set to zero the border-spacing value. There are 
currently many testcases where 'border-collapse: collapse' is improperly 
used.

Also, if border-collapse: collapse must be used, then best is to use 
even border-width values so that fractional pixel situations can be 
avoided.
"
User agents must find a consistent rule for rounding off in the case of 
an odd
number of discrete units (screen pixels, printer dots).
"
17.6.2 The collapsing border model
http://www.w3.org/TR/CSS21/tables.html#collapsing-borders


8-
Avoid single cell with single row tables
****************************************

Mono-cell in mono-table-row table in table layout rendering testcases is 
to be
avoided because those are not very helpful and not very reliable. It may 
lead to false positives. Such are not realistical usages of tables and 
are not typical webpage scenarios either.

eg
http://test.csswg.org/suites/css2.1/20110323/html4/background-position-applies-to-004.htm
is passed by IE9 and Chrome 14 but
http://test.csswg.org/suites/css2.1/20110323/html4/table-backgrounds-bs-row-001.htm
and
http://test.csswg.org/suites/css2.1/20110323/html4/table-backgrounds-bs-row-002.htm
are failed by IE9 and Chrome 14.
Additional testcase: 
http://the-dees.webs.com/iepp1/55-tr-bg-position.html


9-
How to test inline-block
************************
best or better, more recommendable testing of inline-block when its
content does not have any block containers.
e.g.
(bad or less good)
<div>
  <span style="display: inline-block;">text sample</span>
</div>

(good or better)
<div>
  <div style="display: inline-block;">
   <span style="display: block;">text sample1</span>
   <span style="display: block;">text sample2</span>
  </div>
</div>

Minimal code to use with 'display' set to inline-block
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/inline-block-minimal-code-2.html

Baseline-alignment of inline-blocks:
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/vertical-align-baseline-004a.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/vertical-align-baseline-006a.htm


10-
Recommended shape descriptors
*****************************

Rectangle, square, stripe, bar, line, grid.

A stripe should be a short bar. A bar should be a long 1em line. A ideal
rectangle should be 200px wide by 100px tall or 300px wide by 100px 
tall.
An ideal square is 100px by 100px.

The test suite often uses unsuited, unclear terms: "line", "box", 
"block": in the end, there is no reliability, no common sense.

http://test.csswg.org/suites/css2.1/20110323/html4/abspos-001.htm
refers to a "rectangle" when I would use "stripe"

http://test.csswg.org/suites/css2.1/20110323/html4/abspos-024.htm
refers to a "green box" while
http://test.csswg.org/suites/css2.1/20110323/html4/abspos-028.htm
refers to a "green bar".
http://test.csswg.org/suites/css2.1/20110323/html4/background-image-cover-001.htm
refers to a blue box!... when it obviously should be a blue bar.

http://test.csswg.org/suites/css2.1/20110323/html4/c565-list-pos-000.htm
refers to a rectangle when I think it should mention a stripe.

http://test.csswg.org/suites/css2.1/20110323/html4/background-image-005.htm
"This line should be green"... but after a few hundreds of tests, we 
have
no certainty on what "This line should be green" actually means. The 
text
should be green? or the background of the line should be green?

Sometimes, mentioning "a blue square" is sufficient; sometimes, it will 
be
necessary for the tester to know if it's a filled blue square or a white
square with a blue border.

Shape descriptors should be furthermore specified depending on the needs
of a test (from general to more specified):
line vs green line vs vertical green line vs short vertical green line
eg: 
http://test.csswg.org/suites/css2.1/20110323/html4/border-right-applies-to-002.htm

(from general to more specified)
a square, a black square, a filled black square, a filled black
square on the right, etc.
http://test.csswg.org/suites/css2.1/20110323/html4/border-right-color-013.htm

Example
http://test.csswg.org/suites/css2.1/20110323/html4/abspos-001.htm
says "This page should have a single green rectangle below this 
paragraph and no red."
when it would be preferable to say
"Test passes if there is a short green stripe and *no red*."

One danger of overspecifying shape descriptor unneedlessly, causing 
pass/fail sentence to become long and arduous to read:
eg: "short vertical solid continuous line" may be over-doing, 
overdescriptive.

eg How big/large/tall should be a "box"?
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-008.htm
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-009.htm
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-016.htm
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-022.htm
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-023.htm
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-029.htm
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-replaced-height-030.htm

I think we should ban the word "box" in all testcases. For most people, 
a box is 3-dimensional object to begin with.

Reusing terms, expressions used in the general public, in common places 
is
generally good.
Example:
"bathroom tiles" is an excellent expression for use in the test suite.

eg.:
http://test.csswg.org/suites/css2.1/20110323/html4/min-width-036.htm
"Test passes if there is a box below."
but almost any rectangular shape would be a passed test.


11-
Sentences, statements descriptions
**********************************

(good) This sentence should have green text.
(not best, ambiguous) This sentence should be green.
(good) This text should be green.
(bad) This line should be green.
(bad) 
http://test.csswg.org/suites/css2.1/20110323/html4/first-line-pseudo-017.htm
(bad) 
http://test.csswg.org/suites/css2.1/20110323/html4/first-line-pseudo-018.htm
(good) This sentence should have a green background.
Ambiguous: 
http://test.csswg.org/suites/css2.1/20110323/html4/background-image-005.htm
http://test.csswg.org/suites/css2.1/20110323/html4/c531-color-000.htm
http://test.csswg.org/suites/css2.1/20110323/html4/c532-bgcolor-000.htm
(bad) 
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/first-line-001.htm


12-
Removable and emboldenable words
********************************

Removable
=========

visible
viewable
appearing
present
anywhere
displayed
you can see
( 
http://test.csswg.org/suites/compositing-1_dev/nightly-unstable/html/mix-blend-mode-blending-with-sibling.htm 
)
You should see
(http://test.csswg.org/suites/css2.1/20110323/html4/inlines-013.htm)
you can view
in this page
on this page
on the screen
( 
http://test.csswg.org/suites/compositing-1_dev/nightly-unstable/html/mix-blend-mode-blending-with-sibling.htm 
  )
below
below this line (eg 
http://test.csswg.org/suites/css2.1/20110323/html4/replaced-intrinsic-002.htm)
after this line
below this sentence
after this sentence
below this paragraph (eg 
http://test.csswg.org/suites/css2.1/20110323/html4/abspos-001.htm)
under this paragraph (eg 
http://test.csswg.org/suites/css2.1/20110323/html4/tables-002.htm)
in the next paragraph
after this
following
which follows


It is presumed that testers taking the test suite are not blind and do 
not
have an incapacitating daltonism condition or some kind of color vision
illness or incapacity. Mentioning a color implies by itself that it can 
be viewed, it is visible by senses.

eg (bad) "Test passes if there is a green box visible and there is no 
red
visible on the page."
http://test.csswg.org/suites/css2.1/20110323/html4/clip-101.htm

Since 95% (or so) of all tests starts with a sentence stating the
pass and fail condition, then "below", "below this line", "after this 
sentence", "under this paragraph", etc.. is evident, pleonastic.

Why remove such words? Because they are useless, empty-meaning, 
pleonastic or self-evident or irrelevant words; they can be safely 
removed without affecting understanding of pass/fail conditions. 
Anything that shortens the pass/fail conditions sentence of a test, 
without compromising its understanding for a tester, helps. This is 
particularly true if a test suite has 1000 tests or more!

Emboldenable
============

Emboldened words attrack reading, promotes text skimmability, helps text 
scannability, faster test checking. Those words should identify clearly 
and quickly the most important aspect of a pass/fail condition or the 
determinant relationship between 2 objects to compare in the testcase or 
the most determining pass/fail component of a test. All this is 
especially true if a testsuite has over 1000 testcases. So, here's my 
list of emboldenable words:


no red
green square (with relevant shape descriptors: {rectangle, square, 
stripe, bar, line, grid}) [color][shape descriptor]
identical
the same
look the same
in the center
horizontally centered
not or NOT (eg not slanted)
eg not adjoining 
http://test.csswg.org/suites/css2.1/20110111/html4/border-spacing-applies-to-007.htm
filled {rectangle, square, stripe, bar, line, grid}
hollow {rectangle, square, stripe, bar, line, grid}
[color] square with [color] borders
identical colored bands
is in [upper|lower]-[left|right] corner
fills the [upper|lower]-[left|right] corner
inside the square
on the same line
same color
same dimensions
same order
same size
same width
same height
same horizontal position
same vertical position
larger (horizontal axis)
taller (vertical axis)
protude out of
[ top | bottom ] of {rectangle, square, stripe, bar, line, grid} flush 
with
[ top | bottom ] of {rectangle, square, stripe, bar, line, grid} level 
with
[ top | bottom ] of {rectangle, square, stripe, bar, line, grid} aligned 
with
[ no space | no gap ] between
touching
is flush with
is aligned with


Why emboldening such words?
The most important pass conditions are easy to read, to find; this
help eye/reading scannability, text skimmability, faster test checking.

Also removable: "exact" and "perfect" from "exact square", "perfect 
square".
eg exact square:
http://test.csswg.org/suites/css2.1/20110323/html4/replaced-intrinsic-002.htm

"perfectly rectangular"
http://test.csswg.org/suites/css2.1/20110323/html4/blocks-011.htm
http://test.csswg.org/suites/css2.1/20110323/html4/blocks-012.htm
http://test.csswg.org/suites/css2.1/20110323/html4/blocks-014.htm
http://test.csswg.org/suites/css2.1/20110323/html4/blocks-015.htm


13-
"Test passes if there is (are) ..."
***********************************

Most of the testcases now start with "Test passes if there is (are) ..." 
which
is probably the best introductory formulation for testcases; reusing
consistently such formulation makes the reading of testcases more 
predictable.


14-
Horizontal scrollbar
********************

Avoid horizontal scrollbar generated with a window viewport of
1000px wide (screen resolution)
E.g.:
http://test.csswg.org/suites/css2.1/20110323/html4/vertical-align-121.htm


15-
Use description wisely
**********************

http://test.csswg.org/suites/css2.1/20110323/html4/cursor-014.htm
Test passes if a cursor that indicates westward movement appears when 
...
(better) Test passes if a double-headed horizontal arrow appears when 
...


16-
Avoid technical terms, otherwise describe them
**********************************************

(bad) "You should see a 100x100 green square below."
http://test.csswg.org/suites/css2.1/20110111/html4/clipping-002.htm
(bad) "a 10em box"
(bad) "multi-bit alpha channel"
http://test.csswg.org/suites/css2.1/20110323/html4/background-alpha-001.htm


17-
Graphical layout of objects to compare
**************************************

If measurements are performed by comparing with a referring
{stripe, bar, square, rectangle, line}, then best is to align 
horizontallly
the tested geometrical shapes if their height is the object of the test 
and
to align vertically the tested geometrical shapes if their width is the
object of the test.
The general idea is to arrange graphical layout of objects in a way that
eases the task of the tester.


18-
Reuse consistently CSS 2.1 terminology
**************************************

In the text assert and in title, reuse the spec terminology, vocabulary
and do not add new, unknown terminology.
(bad) Shrink-to-fit height
http://test.csswg.org/suites/css2.1/20110323/html4/absolute-non-replaced-height-002.htm
(good) Height is based on content.

"shrink-wraps", "shrink-wrapping" are not defined anywhere in the spec.

"ancestor" is preferable/better to "ascendant" because the spec uses the 
word "ancestor".

"induce a style"
http://test.csswg.org/suites/css2.1/20110323/html4/table-column-rendering-004.htm


19-
Use meaningful, helpful, semantical id or class attributes
**********************************************************

Bad, not recommendable:

(bad) class="class1"
(bad) class="class2"
(bad) class="one"
(bad) class="two"
(bad) class="a"
(bad) class="b"

(bad) id="div1"
(bad) id="div2"
(bad) id="span1"
(bad) id="span2"
(bad) id="one"
(bad) id="two"
(bad) id="a"
(bad) id="b"
id="test"
function test()

Better, more recommendable:

(good) class="abs-pos-parent"
(good) id="rel-pos-grand-parent"
(good) id="containing-block"
(good) class="blue-float-left"
(good) id="red-overlapped"
(good) id="green-overlapping"
(good) id="reference"
(good) id="control"
(good) id="clearing"
(good) id="following-sibling"
(good) class="adjacent-sibling"

Good identifiers are those which are
- descriptive with regards to the working logic of the testcase,
- descriptive of the design logic of the testcase or
- descriptive of the respective position in the containment hierarchy or
- descriptive of the respective position in the positioning hierarchy.

Identifiers should describe the role of an element inside the building
logic of a testcase.


20-
Include calculations and/or explanations into comments of the testcase
***********************************************************************

Eg.

http://test.csswg.org/suites/css2.1/20110323/html4/position-absolute-percentage-inherit-001.htm

http://test.csswg.org/suites/css2.1/20110323/html4/margin-collapse-clear-013.htm


21- Try to make the most important pass/fail condition of a testcase the 
most
important visual aspect of the page and the only/sole visual aspect of 
the page


22-
Sufficient background/foreground color contrast/brightness
**********************************************************

Some tests inadvertedly, awkwardly use 2 dark colors. Or they write with 
a
dark color inside another dark color area.

eg (bad) 
http://test.csswg.org/suites/css2.1/20110323/html4/column-border-002.htm
eg (bad) 
http://test.csswg.org/suites/css2.1/20110323/html4/first-line-inherit-002.htm


23-
Graphically arrange layout to help tester
*****************************************

The testcase should be built in a way that, when it fails, it is 
graphically
clear and obvious that it fails:

In
http://test.csswg.org/suites/css2.1/20110323/html4/block-non-replaced-width-008.htm

http://test.csswg.org/suites/css2.1/nightly-unstable/html4/column-visibility-004.htm

the amount of red is equal to the amount of green. So, in case of a test 
failure, the tester will easily notice the failure.

(bad) 
http://test.csswg.org/suites/css2.1/20110323/html4/max-height-017.htm
An UA which does not implement max-height is going to show 2 squares 
whose
differences of height is going to be 1px only: for many humans, a hardly
noticeable difference. A machine running software would mark this as 
fail
though.


24-
Spelling
********

descendent (eg descendent-selector-00? testcases), PREREQUSITE,
strips (table-borders-002 and 004),


25-
Containing block is not a CSS box
*********************************

The concept of containing block is for positioning; a containing block 
is not a CSS box. A containing block is nothing more than a rectangle 
formed by (see section 10.1 as there are more than 1 answer) which has a 
width, a height, a top edge, a left edge, a bottom edge and a right 
edge. A containing block only serves to position positioned elements. 
So, "padding of containing block" and "content edge of containing block" 
are out of spec terms and should not be used.


26-
Title versus text assert coherence
**********************************

The title of a testcase should never contradict or oppose the assert
text. Coherence, consistence, clarity at all times.

There are *_hundreds_* of testcases where this happens. The title says -
example given
"Fixed position elements impact on later siblings"
but the assert says
"Fixed positioned elements do not impact later siblings."
http://test.csswg.org/suites/css2.1/20110323/html4/position-absolute-001.htm

The thing is that even in the listing of testcases, users will more
likely see the title rather than the assert because of smaller font-size
used for the assert. In any case, a blatant contradiction like this is
never going to help users to understand CSS 2.1. Some people think and
say that CSS 2.1 is tough to read, hard to understand, without enough
examples, etc... and now they could add that the CSS 2.1 test suite does
not help as much as it could... by being consistently clear at all
times.

The title of a testcase should probably only lists the most important
properties or concept involved.

eg:
<title>CSS Test: border-collapse: collapse - adjacent cells with same
border-style and border-width</title>

or

<title>CSS Test: border conflict element - adjacent cells with same
border-style and border-width</title>


27-
Reusing the same rendered layout
********************************

When creating a serie of testcases, try to use and reuse the same
rendered layout. It not only helps the tester but it can reduce the 
number
of reftests needed to create later.

Bad:
Stack of 5 colored bands
------------------------

http://test.csswg.org/suites/css2.1/20110323/html4/margin-collapse-106.htm, 
112: a stack of 5 bands which are from top to bottom: orange, yellow, 
orange, lime, orange

http://test.csswg.org/suites/css2.1/20110323/html4/margin-collapse-110.htm, 
111: a stack of 5 bands which are from top to bottom: orange, lime, 
orange, yellow, orange

http://test.csswg.org/suites/css2.1/20110323/html4/margin-collapse-114.htm: 
a stack of 5 bands which are from top to bottom: yellow, orange, aqua, 
lime, orange

Good:
Stack of 3 colored squares
--------------------------
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/z-index-stack-001.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/z-index-stack-002.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/z-index-stack-003.htm
(albeit dimensions need to be uniformized to 100px by 100px)


28-
Font-size
*********

Try to make test use a reasonable font-size: minimum 16px, recommended 
20px with a 30px leading. eg font: 20px/1.5 serif;

eg.
(less recommendable) 
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/border-padding-bleed-001.htm
(less recommendable) 
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/position-absolute-percentage-inherit-001.htm

(better, more recommendable) 
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/vertical-align-117a.htm
(better, more recommendable) 
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/separated-border-model-007.htm


29-
Consistency of vocabulary, terminology, expressions used, shape 
descriptors
***************************************************************************

box is [color name]
http://test.csswg.org/suites/css2.1/20110323/html4/background-image-cover-001.htm

background of [text] is [color name]
http://test.csswg.org/suites/css2.1/20110323/html4/background-transparency-001.htm

At all times, you want, you aim for a test where pass/fail condition 
sentence is
unequivocal, disambiguated, tries to establish a 1 to 1 (signifier to 
signified) relation.


30-
Unneeded, unnecessary, extraneous declarations
**********************************************

width: 0
--------
If an element applies shrink-to-fit width (like a table cell), then 
setting its
width to 0 is unneeded and unnecessary.

http://test.csswg.org/suites/css2.1/20110323/html4/border-right-color-applies-to-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-right-color-applies-to-004.htm

An empty cell will not under normal circumstances require that its box 
occupy/creates an horizontal space for its empty content. An empty cell 
that has 'width: auto' will use/occupy/generate a zero width box, a 
content box of 0 width.

table-layout: fixed
-------------------
In order to trigger the fixed table layout algorithm and to get some 
kind of layout effect out of it, some kind of rendering effect out of 
it, the table width or at least one column width or at least one table 
cell width in first row has to be specified.
If none of those 3 is specified, then 'table-layout: fixed' does not and
can not have any useful or meaningful rendering effect.
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-004.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-006.htm
etc.

margin-left: 1in
----------------
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-002.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-003.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-004.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-005.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-006.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-007.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-013.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-left-width-applies-to-014.htm

In the 'border-collapse: collapse' model,

"
The left border width of the table is half of the first cell's collapsed
left border, and the right border width of the table is half of the last
cell's collapsed right border. If subsequent rows have larger collapsed
left and right borders, then any excess spills into the margin area of
the table.
"
http://www.w3.org/TR/CSS21/tables.html#collapsing-borders

There is no subsequent rows in those tests, so no possible excess 
spilling. So, I see no reason to set a margin-left to 1in for these 
tests.

Where exactly the left border of a table starts in comparison to where
its content box starts is a distinct and separate issue which does not 
and should not affect where the black border starts being painted in all 
those tests.

http://lists.w3.org/Archives/Public/public-css-testsuite/2012Jun/0000.html

height: 0
---------
http://test.csswg.org/suites/css2.1/20110323/html4/border-bottom-width-061.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-bottom-width-062.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-top-width-061.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-top-width-062.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-002.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-003.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-004.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-005.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-006.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-width-007.htm

Under normal circumstances, an empty div does not require height: 0. 
Under
normal circumstances, an div with 'height: auto' which does not have any 
content
nor children will occupy/generate a height of 0.

height: 1in
-----------
http://test.csswg.org/suites/css2.1/20110323/html4/border-top-width-092.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-top-width-093.htm
http://test.csswg.org/suites/css2.1/20110323/html4/border-top-width-094.htm

Removed because pointless, irrelevant to the tests.

width: 1in set to an inline non-replaced
----------------------------------------
http://test.csswg.org/suites/css2.1/20110323/html4/box-generation-001.htm
http://test.csswg.org/suites/css2.1/20110323/html4/box-generation-002.htm
Setting a width to an inline non-replaced element has no effect, has no 
rendering or layout effect.

border-collapse: collapse with border-spacing: 0
------------------------------------------------
border-spacing does not apply in border-collapse: collapse model. So, 
just border-collapse: collapse is and should be sufficient.

* {margin: 0}
-------------
This is not suitable, not needed and not recommendable in tests.


31-
1px difference
**************

div#test {max-height: 72pt; height: 73pt;}
div#control {height: 72pt;}
(...)
<p>Test passes if the two boxes below are the same height.</p>

Example taken from
http://test.csswg.org/shepherd/testcase/max-height-017/#comment-c3e1d615f21e
http://test.csswg.org/suites/css2.1/20110323/html4/max-height-017.htm


32-
Single character testcases
**************************
Testcases which involve comparison of 2 characters or which involve a
single character should be using a minimum font-size of 20px or even
bigger (say 32px or 64px).
Some test pass condition involve the color of a single character.
E.g.:
http://test.csswg.org/suites/css2.1/20110323/html4/first-letter-selector-001.htm
and there are many first-letter testcases
E.g.:
http://test.csswg.org/suites/css2.1/20110323/html4/character-representation-001.htm
should have the 2 characters at least 32px.

Anything that help testers, reduce condition of mistakes, speed their
tasks, etc. is better.


33-
Miscellaneous
*************

- Some test create a vertical scrollbar (not sure why or if this is part
of test). E.g.:
http://test.csswg.org/suites/css2.1/20110323/html4/floats-rule3-outside-left-001.htm
- File numbering: should always start at 001, not at 000: that would 
make
counting much easier. E.g.: property-variable-001.xht has to be the 
first
testcase of a serie of testcases with that same base filename.
- Tests with a font-size under 16px. Unless the test goals involves,
requires testing small font-size, all tests should be using the browser
default font-size of 16px.


34-
Better color descriptors
************************

Lime or bright green or light green is not green.
eg
http://test.csswg.org/suites/css2.1/20110323/html4/inlines-020.htm

Navy blue or navy is not blue.
http://test.csswg.org/suites/css2.1/20110323/html4/blocks-011.htm
http://test.csswg.org/suites/css2.1/20110323/html4/blocks-012.htm


35-
Standardized color for the pass/fail conditions sentence
********************************************************

Some Ian 'Hixie' Hickson tests use unneedlessly blue navy for the 
pass/fail conditions sentence.


36-
Too long
********

Filename too long:
http://test.csswg.org/suites/css2.1/20110323/html4/position-absolute-percentage-inherit-001.htm

Test is maybe too long:
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/font-family-rule-004a.htm
That test should maybe be split... on the other hand, the font to be 
loaded will be good for all those 11 sub-tests.


37-
Short but clear pass-fail conditions sentence
*********************************************

Ideally, you want a short but clear pass-fail conditions sentence: the 
shorter, the better. Less reading; less thinking, less prone to 
interpretation.

http://www.gtalbot.org/BrowserBugsSection/css21testsuite/cursor-025.html
had 146 characters:
"
Hover the pointing device inside the rectangle. Test passes if the 
pointing device becomes a small bright green square with the word "PASS" 
in it.
"

and it was changed to:

"
Test passes if the cursor becomes [PASS] when inside the rectangle.
"
which has 60 characters.


38-
Testing 'inherit' keyword on properties that inherit by default
*******************************************************************

Tests that check 'inherit' keyword on properties that inherit by default 
should be removed from test suites. Testing with 'inherit' keyword 
should only be done on properties that do not inherit by default.

So:
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/list-style-type-016.htm
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/font-size-112.htm 
(this one even is affected by a specificity issue)
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/caption-side-003.htm
etc.
would be removed.


39-
An ideal test is ...
********************

An ideal test is one that has a very narrow corridor of success and very 
large corridor of failure.

Eg.:
http://test.csswg.org/suites/css-backgrounds-3_dev/nightly-unstable/html4/background-size-009.html
at line 22:
I changed 'height: 100px' to 'height: 50px' to "enlarge" the corridor of 
failure.
More info: 
http://test.csswg.org/shepherd/testcase/background-size-009/#comment-2f7643da4c58

http://test.csswg.org/suites/css-backgrounds-3_dev/nightly-unstable/html4/background-size-014.html
at line 24:
I changed 'width: 200px' for 'width: 50px' to "enlarge" the corridor of 
failure.
More info:
http://test.csswg.org/shepherd/testcase/background-size-014/#comment-f2d2f913b910


An ideal test is one that is very demanding, requiring.

An ideal test is one that checks, verifies one and only one single 
aspect of the spec.

Anything that simplifies the task of testers (people taking the tests) 
is, as a general rule, welcomed, better, more desirable.

Maintaining a moderate skepticism, vigilance toward browser 
manufacturers and implementors is a sane and cautious test 
creation/review attitude to adopt; "Trust no browser!" means to not 
develop tests with browsers, not develop tests alongside browsers but to 
develop tests thanks to and starting with only or mostly with your 
understanding of the spec.

--------------

Friday August 21st 2015

ref-black-96px-square-no-red  : referenced by  12 tests
ref-filled-black-96px-square  : referenced by 180 tests
ref-filled-green-100px-square : referenced by 319 tests (5 changesets! + 
40 more if at-supports-001-ref.html would be modified a bit.)
ref-if-there-is-no-red        : referenced by 289 tests (2 changesets!)
ref-no-vert-space-between     : referenced by 188 tests
ref-nothing-below             : referenced by  22 tests
ref-this-text-should-be-green : referenced by   6 tests
========================================================
                                              1046 tests

Friday August 21st 2015


-- 
Writing Tests
http://web-platform-tests.org/writing-tests/

General Test Guidelines
http://web-platform-tests.org/writing-tests/general-guidelines.html

Test Review Checklist
http://web-platform-tests.org/reviewing-tests/checklist.html

Test Templates
http://web-platform-tests.org/appendix/test-templates.html

Writing Reftests
http://web-platform-tests.org/writing-tests/reftests.html
Received on Monday, 11 June 2018 22:27:30 UTC

This archive was generated by hypermail 2.4.0 : Friday, 20 January 2023 19:58:22 UTC