RE: Comments on Microsoft Chapter 4 Tests


> Anne van Kesteren wrote:
> > On Wed, 18 Mar 2009 20:00:48 +0100, Grant, Melinda wrote:
> > Assuming others agree that in general this is a good change (it seems
> so
> > to me), I think the text needs to be more descriptive of the expected
> > output, in this case,
> >  <p>This text should be green.</p>
> 
> That works.
> 

That doesn't work for me. We should avoid using any of the special words (MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY and OPTIONAL) in pass conditions. These all have special meanings and should be used sparingly and only when it is really appropriate. In fact I think I only use any of those terms maybe 10 or so times throughout the entire test suite and only when it is left up to the user agent to decide what to do. Also the example <p>This text should be green</p> is flawed. It's not that is "should be" it must be green to pass. How do you know if the test passes you have no check? You need to state what needs to happen in order to pass the test. "Test passes if" was our standard and clearly tells anybody when the test passes.

Also changing the pass conditions to also be part of the test can lead to testing issues. We originally had the tests in this combined format but after writing 10000+ tests we discovered that you have to keep the tests consistent for running the tests quickly. When you combine the test and the pass conditions you can get unexpected behavior that makes the pass conditions hard to read or they are completely unreadable. In that situation do you still pass the test? You don't know because you can't read the pass conditions. The test might actually pass if you hadn't screwed up the pass conditions. The separate pass condition leaves the least room for error when running the cases based on the feedback and running of the cases within our team. Changing the tests just makes it slower to run the cases and potentially causes problems.

> 
> > In many existing tests, it's not clear without examining the source
> > whether the text should be green or whether it should have a green
> > background, whether the background should extend across the page,
> etc.
> > I've had testers fail tests that use the "This line..." description
> > because there is no line on the page...
> >
> > We need to be more explicit, ala,
> >
> >  <p>This sentence should have a green background which extends
> across
> > the display area.</p>
> >  <p>This paragraph should be inside a green rectangle.</p>
> >  <p>This sentence has a solid green line beneath it which spans
> spaces
> > between words.</p>
> >
> > Etc.
> 
> I suppose we could try to do that though in general it seems easier to
> educate the testers that the expected result is either green or red and
> that whether it is a text or background color should not matter much.
> 

It is important to clearly state what your tests need to do to pass. If they aren't then the tests need to be fixed. Tester education isn't possible in all situations the more clear and understandable the pass conditions are the better. If you are supposed to have a green box then you need to say green box. But if the background of that box is what's important then you need to state that in a way that makes it clear that the box is green. There are a few standard pass conditions that all of our tests follow. You will see the pattern quite quickly when you run the next release of the test suite. All tests have been cleaned up to use standard text and have a consistent format. It makes running 7000+ tests by hand (which is the only way to run them off the test suite site at the moment) very easy.

We have many standard texts when running the tests:

Test passes if there is no red visible on the page.
Test passes if the box below is green.
Test passes is the "Filler Text" below is green.
Test passes if there is a dotted green line below.
Test passes if the line below is green.
Test passes if the blue box below is to the right of the orange box.

None of these examples of standard text leave any room for error they clearly state what needs to be true to pass the test. I know we are all smart people and we may think this is dumb to have things so clearly stated and maybe it even complicates the test a little but we do not know the intelligence level or skill that every person running theses tests will have. The simpler and the more clearly things are stated more people will be able to run them and verify compliance. As it stands now with our tests a tester does not need to run the tests, anyone can run the tests and clearly understand if they pass or fail and 99% of the time they can say why they fail too.

As a note it is probably best not to use the term background in a test. Or any other term that is a direct property reference. I know some of our cases do but I am still trying to clean those up. Again the tests have to be simple and the more technical the term the harder it is for anyone to run the tests.

Received on Wednesday, 18 March 2009 21:42:03 UTC