Feedback and suggestions on CSS2.1 Test Case Authoring Guidelines

Hello all,

These are some thoughts I have come up with after carefully examining at
least 1,000 testcases so far from the test suite.

I have already provided some feedback here:

General and specific comments, feedback on Microsoft submitted tests
http://lists.w3.org/Archives/Public/public-css-testsuite/2009Sep/0014.html



Suggestions and recommendations for CSS2.1 Test Case Authoring
Guidelines

1- The title of a testcase should never contradict or oppose the assert
text. Coherence, consistence, clarity at all times.

There are *_hundreds_* of testcases where this happens. The title says -
example given -
"Fixed position elements impact on later siblings"
but the assert says
"Fixed positioned elements do not impact later siblings."

The thing is that even in the listing of testcases, users will more
likely see the title rather than the assert because of smaller font-size
used for the assert. In any case, a blatant contradiction like this is
never going to help users to understand CSS 2.1. Some people think and
say that CSS 2.1 is tough to read, hard to understand, without enough
examples, etc... and now they could add that the CSS 2.1 test suite does
not help as much as it could... by being consistently clear at all
times.

2- Meaningful, intuitive, descriptive, specific and helpful identifiers

Good:
id="containing-block"
id="wrapper"
class="red-floated-left"
id="overlapping-green"
id="rel-pos-container"
id="parent"
id="following-sibling"
id="grand-parent"
id="rel-pos-parent"
id="cleared-both"


Bad (otherwise not ideal):
id="test"
class="class"
function test()
id="a"
id="b"
class="one"
class="two"
id="div1"
id="div2"
id="span1"

Good identifiers is when
- descriptive with regards to the working logic of the testcase,
- descriptive of the design logic of the testcase or
- descriptive of the respective position in the containment hierarchy or
- descriptive of the respective position in the positioning hierarchy.

Identifiers should describe the role of an element inside the building
logic of a testcase.


3- Avoid reusing the same identifiers in the same testcase

I have seen this in a testcase:
<body onload="setTimeout(test, 0)">
<table class="test">
(...)
<div id="test">

See:
http://test.csswg.org/suites/css2.1/20100727/html4/tables-101.htm

.a { text-decoration: underline; }
#a { font-style: italic; }

See:
http://test.csswg.org/suites/css2.1/20100727/html4/c13-inh-underlin-000.htm


4- Avoid inventing new words or new expressions which are not used by
the CSS 2.1 itself, furthermore if the CSS 2.1 spec itself does not
define it

E.g.: "shrink-to-fit height", "shrinks-to-fit height", "shrink-wraps",
"shrink wrapping", etc.. are used at least 50 times in the test suite
but actually are never defined anywhere at CSS 2.1 spec.

There should be a strict correspondence of terminology used between the
test suite and the spec. This is furthermore important for people who
may not be fluent in english or those who do not master english.

http://test.csswg.org/suites/css2.1/20100727/html4/absolute-non-replaced-height-002.htm

"
 absolute-non-replaced-height-002
 Absolutely positioned, non-replaced elements, shrinks-to-fit height

    * The used value for absolutely positioned elements shrinks-to-fit
height.
    "

but the spec says "height is based on the content"; the spec never
speaks of shrinks-to-fit height and shrink-wrapping.

Same thing with
http://test.csswg.org/suites/css2.1/20100727/html4/absolute-non-replaced-height-007.htm
http://test.csswg.org/suites/css2.1/20100727/html4/absolute-non-replaced-height-009.htm
http://test.csswg.org/suites/css2.1/20100727/html4/absolute-non-replaced-max-height-002.htm

etc.

Sometimes, reading a technical spec (written in english) is tough enough
for a lot of amateur web authors who, on top of that, may not have
english as their mother tongue... so, creating new expressions should be
avoided. Blindfully, automatically, mechanically reusing the same
terminology, vocabulary is the safe and sound thing to do.

5- Always try to use distinguishible content - whenever possible - so
that examining the source code can be easier, so that the efforts of
understanding the concepts - sometimes difficult ones - and the testcase
itself (and its complexity) can be less demanding

E.g.:

<p>x <span>xxxxx xxxxx</span> x</p>

versus

<p>a <span>ceikl mnors</span> u</p>

Today, the browser web inspecting tools we use (IE8's Developer tools,
Firefox's DOM inspector, Opera's DragonFly, Webkit's Web Inspector,
Konqueror's DOM treeviewer) are not very powerful to deal with inline
elements... but one day they might or one day someone could write an
extension for handling, inspecting inline elements ... in which case,
having to debug inline content like contiguous "x"es is not fun.

References:
http://test.csswg.org/suites/css2.1/20100727/html4/c5502-imrgn-r-001.htm
versus
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/c5502-imrgn-r-001-gt.html

6- Provide calculations and relevant explanations in the source code of
complex examples

E.g.:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/left-offset-percentage-002.html
or
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/margin-percentage-inherit-001.html
or
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/margin-em-inherit-001.html

7- Try to write comments, explanations, assert which may assist the
tester who may be a beginner in CSS.

- give calculations of clearance being done
eg:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/margin-collapse-clear-012.html

- give resulting calculations of margin collapsing
eg:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/margin-collapse-clear-017.html

- give calculations about positioning involving relative unit: em, %,
inherit
eg:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/margin-percentage-inherit-001.html

8- Avoid creating rules and declarations which are unneeded in the
testcase or which may in fact even interfere with the purpose of the
testcase.

Ideally, a testcase should only use what is absolutely required, needed
for a testcase to perform, to test what it is supposed to test. It
should be strictly aiming, targeting at the purpose of the testcase
itself.
Direct. Terse. Spare. Straightforward. No extraneous elements. No
unneeded containers. No unnecessary classes. No kidding.

Example given:

http://test.csswg.org/suites/css2.1/20100727/html4/margin-inline-001.htm

uses, declares:

             div
             {
                 background: red;
                 border: solid green;
                 font-family: ahem;
                 line-height: 1;
                 width: 4em;
             }
             span
             {
                 background: green;
                 color: green;
                 margin: 2em 0;
             }
        </style>
    </head>
    <body>
        <p>Test passes if there is a green box below and no red.</p>
         <div>
             <span>XXXX XXXX XXXX</span>
         </div>

when, in all fairness, it should only declare:

             div
             {
                 background: red;
                 font-family: ahem;
                 line-height: 1;
                 width: 4em;
             }
             span
             {
                 color: green;
                 margin: 2em 0;
             }

9- When it fails, it should fail [ideally] clearly, in a way that the
user can unequivocally see a failure.

Ideally, the testcase should provide as much red (when the testcase
fails) as there is green when the testcase passes when the overlapping
technique is involved. Same thing with a ruler: if the gap is
supposed to be 100px when the testcase passes, then you want the gap to
be twice as big when the testcase fails. This is the principle I tried
to follow in all my own testcases [1].

Here are one example of where a general, lenient pass condition actually
hide a test failure:

http://test.csswg.org/suites/css2.1/20100727/html4/absolute-non-replaced-height-001.htm

In Chrome 5.0 and Safari 5.0.1, there will be a small gap between the
blue square and the black borders. Lots of people will (or may) think or
assume that the test passes in Chrome 5.0 and Safari 5.0.1 but, in fact,
the test awkwardly reveals a failure in implementing top and left auto
for fixed elements ... which was the purpose of the testcase!

http://www.gtalbot.org/BrowserBugsSection/css21testsuite/absolute-non-replaced-height-001-gt1.html
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/absolute-non-replaced-height-001-gt2.html
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/absolute-non-replaced-height-001-gt3.html
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/absolute-non-replaced-height-001-gt4.html
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/absolute-non-replaced-height-001-gt5.html
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/absolute-non-replaced-height-001-gt6.html

demonstrate this.

[1: This principle may be difficult to implement. In some of the
testcases I submitted, when tried with older browsers like Netscape 7.2,
some testcases will not show any red but the test nevertheless fails in
NS 7.2. It's impossible to predict how a testcase can be failed in
various browsers. But this should not deter a practice trying to provide
strict and requiring pass conditions.]

10- Minimum font for testcases should be 16px, not 10px unless the
testcase require to test for smaller font-size. For lots of reasons. For
tons of reasons!

E.g:
http://test.csswg.org/suites/css2.1/20100727/html4/c5501-mrgn-t-000.htm
versus
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/c5501-mrgn-t-000-gt.html

regards, Gérard
-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 test suite (beta 2; July 27th 2010):
http://test.csswg.org/suites/css2.1/20100727/html4/toc.html

CSS 2.1 test suite contributors:
http://test.csswg.org/source/contributors/

Received on Sunday, 1 August 2010 22:07:43 UTC