Some proposals on Test Review Checklist and Test Style Guidelines

Rebecca,

1-

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

I believe this page may be source of confusion.

"All tests" is supposed to mean a) non-self-describing tests and b) 
self-describing tests;  each ( a and b ) categories of tests could be of 
type manual (not a reftest) or automatable (a reftest).

When you refer to "Reftests", I believe you mean tests that have an(or 
several) associated reference file(s). But this may not be what people 
would think...

Here's my proposal:

{

All tests
---------

The test passes when it's supposed to pass.

The test fails when it's supposed to fail.

The test is testing what it thinks it's testing.

The spec backs up the expected behavior in the test.

The test is automated as either reftest or a script test unless there's 
a very good reason why the test must be manual.

The test does not use external resources.

The test does not use proprietary features (vendor-prefixed or 
otherwise).

The title is descriptive but not too wordy.

The test is as cross-platform as reasonably possible, working across 
different devices, screen resolutions, paper sizes, etc.


Self-describing tests
---------------------
The self-describing statement is clear, short and self-explanatory. Your 
mother/husband/roommate/brother/bus driver should be able to say whether 
the test passed or failed within a few seconds, and not need to spend 
several minutes thinking or asking questions.


Reference file only
-------------------
The reference file is accurate and will render pixel-perfect identically 
to the test on all platforms.

The reference file uses a different technique that won't fail in the 
same way as the test.


Script Tests Only
-----------------
no change; same as now

In depth Checklist
------------------
no change; same as now

}

Note that I am also proposing some changes.
"The self-describing statement is accurate, precise, simple, and 
self-explanatory."
is now
"The self-describing statement is clear, short and self-explanatory."

Also, I propose to remove
"If there are limitations (e.g. the test will only work on 96dpi 
devices, or screens wider than 200 pixels), then these are documented in 
the instructions."

2-

If a test is supposed to only work in a 96dpi device or paper media only 
or etc.., then test creators only need to use accordingly this list
http://testthewebforward.org/docs/test-templates.html#requirement-flags

3-

This example
http://testthewebforward.org/docs/reftests.html#css-example
coming from
http://test.csswg.org/suites/css2.1/nightly-unstable/html4/border-bottom-applies-to-009.htm
is not best (and that's my fault!) because green is used without red in 
case of a failure. I would need to change those 
border-bottom-applies-to-* tests so that they would use
border-bottom-width-applies-to-* tests. At the same time, they would 
reuse the same reference files.

4-

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

a)
http://testthewebforward.org/docs/test-style-guidelines.html#indicating-success
"This line should ..."
I am for replacing "line" by "sentence" or by "text" and to keep "line" 
for linear test situations.
I am also strongly for systematic usage of the "Test passes if ..." 
introductory words in every examples of self-describing sentences.

b)
http://testthewebforward.org/docs/test-style-guidelines.html#indicating-success
I am for replacing
"Test passes if there is a green square and no red."
with
"Test passes if there is a filled green square and *no red*."
because
ref-filled-green-100px-square
is already referenced by 149 tests and there is no reason why it would 
not or could not be referenced by thousands of tests.

c)
" on this page"
in self-describing sentences can be safely removed. Same thing with
"visible"
"viewable"
"appearing"
"present"
"anywhere"
"displayed"
"you can see"
"You should see"
"you can view"
"in this page"
"on this page"
"below"
"below this line"
"after this line"
"below this sentence"
"after this sentence"
"below this paragraph"
"under this paragraph"
"in the next paragraph"
"after this"
"following"
"which follows"

If self-describing tests all start with the recommended "Test passes if 
..." and if testers are assumed to not be blind, then all these 
expressions can safely be removed.

d) "Filler text" should be preferred for page background; "Text sample" 
should be preferred when a text is being the object of the test.


Gérard
-- 
Web authors' contributions to CSS 2.1 test suite
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html
CSS 2.1 Test suite RC6, March 23rd 2011
http://test.csswg.org/suites/css2.1/20110323/html4/toc.html

Received on Tuesday, 15 April 2014 10:57:00 UTC