Re: Better Test Case Coding for Reviewers

Le 2015-07-05 12:31, fantasai a écrit :
> [CCing m.d.platform, since it might be helpful for layout tests there, 
> too.]
> 
> I've been reviewing some tests lately, and there are three things that
> would help a lot in a correct and efficient review.
> 
> #1: Good indentation.
> 
>   The contents of each block should be indented. It's much harder to
>   read the style sheet if this is not true. I've seen quite a few tests
>   that have this problem. :(


I am for this. We need to give an example of recommended indentation:

   <style>
   p
     {
       margin: 1em 0em;
     }
   </style>

and the documentation itself should also start giving the example, 
practicing its very own recommendations.

> #2: Separating framework from variation.
> 
>   Most tests have code that sets up a framework to reveal the results
>   of a test, and then code that sets up the specific situation being
>   tested. The framework does not vary from test to test within a set,
>   but the test declarations do.
> 
>   For example, in a test for background positioning, there will be
>   rules to set up a box, e.g. width, height, border, padding, margins,
>   color, background-repeat, etc. These are the framework declarations.
>   They do not vary by test within the set. Then there will be rules
>   that set up the test, e.g. background-position, background-origin,
>   background-size. These vary by test.
> 
>   If there are gratuitous differences in formatting, ordering,
>   indentation, of the framework declarations, or if the test mixes
>   the framework declarations with the test declarations, then I have
>   to review the entire framework for every test in the series, because
>   I'm not sure what's changed.

I wish we had this kind of email discussion years ago. Before CSS2.1 
test suite even started. Especially before thousands of tests get 
created.

>   However, if the framework declarations are kept constant, and the
>   test declarations -- the part that varies per test in the series --

I think such coding guideline should be even followed by tests that are 
not in a serie of tests. That way, all tests would be a bit easier to 
review.

>   are pulled out and isolated into their own part of the style sheet,
>   then I can quickly review an entire series, and I am less likely to
>   make mistakes in the review.
> 
>   For example, for test file 1
> 
>     /* Framework */
>     .container {
>       border: solid 3px 5px 7px 2px;
>       padding: 11px 17px 19px 13px;
>       margin: 1em;
>       background-image: url(support/swatch-green.png);
>       background-repeat: no-repeat;
>     }
> 
>     /* Test */
>     .container {
>       background-position: center;
>     }
> 
>   and test file 2
> 
>     /* Framework */
>     .container {
>       border: solid 3px 5px 7px 2px;
>       padding: 11px 17px 19px 13px;
>       margin: 1em;
>       background-image: url(support/swatch-green.png);
>       background-repeat: no-repeat;
>     }
> 
>     /* Test */
>     .container {
>       background-position: left;
>     }
> 
>   I can quickly see, at a glance, that the framework has not changed.
>   And that the only difference in the two tests is that 'center' was
>   replaced with 'right' in the background-position rule.

You most likely meant to say 'left' in that last sentence. :)

> 

To a large extent, Richard Ishida's tests makes this kind of code 
separation, of code identification.

eg:
http://test.csswg.org/source/css-writing-modes-3/bidi-embed-002.html

The more we do tests review, the more we can see+understand what would 
ease, would help reviewing, would make reviewing more efficient, 
reliable, etc... But a wide majority of test authors are not reviewers 
and do not see any importance or relevance of to such details.

Creating one single test that would meet all possible criteria, 
recommendations can take a lot of time and a lot of reading.

There are many other coding habits or coding guidelines that can help 
reviewing, can help make tests more reliable, less confusing, make the 
task of testers easier, faster, etc... Taking the 800 tests of a test 
suite with a browser takes time ... And reusing the same reference file 
reduces the load of the server.

> #3: Declaration of intent
> 
>   If the test writer tells me what they're trying to test, 
> specifically,
>   in this test, rather than having me guess from the source code, I'm
>   much more likely to detect when they're failing at their job, either
>   by flat out not triggering the right situation at all, or by not
>   checking some modes of failure. (I see both of these happen often;
>   it is not a theoretical problem.)

Both the <title> and the meta assert text should be doing all that.

Sometimes, the multiple directives, requirements can overconstrain the 
recommendation itself: eg the title should be short and descriptive... 
but sometimes, this is just impossible to have both.

Gérard

>   The CSSWG tests have a <meta name="assert"> field for this purpose,
>   but, really, a genuinely useful comment in any format would help.
>   We comment our functions to say what they're supposed to do, but
>   for some reason we don't think it's necessary to do this for tests.
>   Non-trivial tests should have a comment stating their raison d'être.
> 
> ~fantasai

-- 
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Received on Sunday, 5 July 2015 20:42:31 UTC