Thoughts On Combining Tests

[Some thoughts on the topic, from the perspective of a reviewer...]

On the one hand, putting too many tests in one file can make it more complex,
and harder to keep track of what's being tested.

On the other, splitting tests into too many files can make it more complex,
and harder to understand what's being tested.

====

It's not always a good idea to split things out as much as possible.

For example, a test for the effect of 'writing-mode: vertical-rl' changing
the block flow direction would need to test many things: that blocks stack
right to left, that lines stack right to left, that table rows stack right
to left, that the table caption is to the right of the table, that lists
stack right to left, that list markers are at the top of the box, not its
side, that adjacent horizontal margins collapse, etc.

All of these should not go into one test. It would be a mess: either a
collection of random unrelated things or a very tangled up testcase, or
a mixture of both.

However, some of these are better off placed into the same test:

   * a single <div> can hold a lot of different block structures
     - sibling child blocks
     - sibling child line boxes
     - block-in-inline splits
     and ensure that they all flow in the correct order, just as
     easily as it can test any one of these

   * a single <table> can hold a lot of different table structures
     - table-header-groups
     - table-footer-groups
     - table-row-groups
     - colspanned cells
     - rowspanned cells
     and ensure that they all flow in the correct order, just as
     easily as it can test any one of these

We could split these out into many tests, one for each thing listed,
but that would not make them easier to understand, because they are
not orthogonal or intersecting considerations but parallel objects
that can be tested as a list against the test code.

Splitting them out actually makes it harder to understand, because
there are now many files that repeat a lot of testing code, whereas
in one file they can share the setup code.

Trying to cram as much as possible into one file is not good.
Trying to split as much apart as possible is also not good.
The goal of splitting files or keeping them together should be
understandability--not just of any individual test in isolation,
but of a related series when considered as a whole.

~fantasai

Received on Sunday, 5 July 2015 18:14:38 UTC