Archive for June, 2009

Back To Basics With Integration Testing

Wednesday, June 3rd, 2009

A lot gets written about unit testing and test driven development these days. However I feel integration testing get overlooked sometimes. Everybody seems to know what integration tests are but very few seem to know how to write a good integration tests or  really what are the true differences between integration and unit tests are.

Integration Tests Masquerading As Unit Tests

One of the most common things I see in the industry today is developers writing unit tests that are in fact integration tests. The giveaway signs are batch files copying configuration files,  connection strings and urls to web services all over the place. The tests pass on one developers machine but fail on an others. When you try to set up the solution on your machine it takes you half a day to get the tests to pass. Does any of this sound familiar?

In my opinion these are the worst kind of tests. The tests themselves tend to be very fine grained. They might be trying to test a very discrete portion of code in the same way that a unit test might. But because the code we are testing has so much freedom to access so much of our system. The set up for our test, to put our system into the correct state, in order to execute the code we wish to test is huge.

The problem with these type of tests are that they are incredibly hard to maintain. The fact that this piece of data must be in this table and this web service must do such and such when we call it in order to make our test pass may have made sense to the developer who wrote the code at the time, but for the guy trying to maintain that test six months down later the task can sometimes feel like a Sherlock Holmes mystery trying to fathom out why that test is failing and  how to get it to pass.

What is a Unit Test

From Wikipedia “In computer programming, unit testing is a method of testing that verifies the individual units of source code are working properly. A unit is the smallest testable part of an application.”

In C# a unit test should be aimed at testing the logic at the class level and that classes interactions with the other dependent objects within your system. That means I don’t really want my unit test to be interacting with any databases, web services, smtp servers, file systems etc.. The reason why is that I want my unit tests to be small, fast, robust, simple and easy to maintain. I don’t want to have write 500 lines of code to set up my system to allow my code to execute. I want to control the class under test’s interactions with the other objects in the system in a controlled  and repeatable manner.

When writing a unit test most of the time I am only concerned with the logic contained within that class. I’m looking at writing tests that can exercise as much of the code, within that class and to to execute it in as many different possible ways in order to find those subtle and hard to find bugs. How does the code behave if this or that happens. My unit tests will certainly include the happy logic path but they will also include the unhappy path through my code as well. It is with my unit tests that I can determine how my code will behave in those unusual circumstances. And it is this process of looking at your code under a microscope that helps you build that robustness into your code at the building block level.

To have the freedom to execute our class under test in anyway I choose. The easiest way is to isolate our class from its external dependencies. This is where the tricks of the trade like Inversion of Control and mocking frameworks come in useful. For example if our code needs to make a call of to our data access layer to in order to get a list of customers we can hide our data access dependency away behind an abstraction.Which can be replaced with a mock object that has the desired behaviour we need to satisfy our test and inject it into the class we wish to test.

What Is An Integration Test

So unit tests are great, you can have 100% code coverage but sometimes when you look down the microscope you miss the big picture. Have you set those bindings up correctly on that WCF endpoint? or has the database been set up correctly?

The key to a good integration test is in the name – your testing how all those different components in your system integrate together. Now when I’m writing an integration test I’m not really thinking about uncovering those devious hard to find bugs but trying to test if my system works as expected. I’m looking for those blaring obvious problems, and testing a few of the main logic paths through my application.

I would argue that integration tests have the following characteristics:

  • They are slower to run than unit tests.
  • They are generally harder to maintain than unit tests.
  • They require more code to set them up.
  • They are generally easier to write than unit tests (they need less specialist knowledge in TDD)
  • Can find problems that would be impossible to detect with unit tests.

Given these characteristics you typically don’t want to writing too many integration tests. I remember having a discussion with a colleague a few years ago who argued that integration tests were a far more efficient use of time as you could test a lot more code with far fewer tests – more bang for your buck.

He wasn’t wrong but he missed the point totally. With an integration test it is very difficult to test all your code in your system, such as those hard to reach if statements or that rarely used catch block. Rob Connery blogged the other day about the fact that 80% of all bugs was caused by 20% of code. The risk of not having tests over that code could cause you major problems in the long run. A few integration tests can provide you with a pretty good code coverage figures for very little effort. The problem is to increase that code coverage using integration tests alone the effort involved in writing and maintaing those tests goes up exponentially.

What Makes A Good Integration Test

  • Optimize – Integration tests are never going to be as fast as unit tests but think about ways you can optimize them to run faster. If they take less time your more likely to run them more often.
  • Automation – Make the tests as self contained look at automating as much as possible if the integration test runs against a web service look at ways of spinning up the web service from the test rather than relying on the developer having to install something. The ultimate goal is that a developer should be able to set up the solution from source control and run all the unit tests and integration tests with minimal environmental setup.
  • Data Creation – This might seem like pretty basic stuff but I see people forget to do this all the time. If an integration test needs a certain pieces of data for it to work, don’t expect that data to exist. If you need it for your test to pass make sure your test creates it. This might seem like more work but I’ve seen many an integration test retired because nobody has got the time to work out what piece of data is missing to get it working again. I’ve found LinqToSql to be very handy in the past to create and even verify data exists in a pretty quick and dirty way.
  • Clean Up – If you create test data in order for your test to run, especially if that data could influence other tests then clean up after yourself. Only the other week I wasted a lot of time trying to work out why an integration test was failing – which resulted in some old bit of data that shouldn’t  have been there.
  • Maintenance – Think about writing that integration test in such a way so that that test can be maintained by others. There’s no point investing a lot of time an effort into writing an integration test that somebody a few months later doesn’t understand. All that will happen is it’ll get commented out, deleted or fixed up simply too make the test pass without any understanding of what it’s doing. Make sure that test is still adding value in six months time.
  • Comments – Integration tests are never gonna be pretty. Sometimes its hard to see what an integration test is really trying to test. Make sure your integration test has a good set of comments with it.
  • One Assertion Per Test – In just the same way as when writing unit tests limit your test to one assertion per test. When there are multiple assertions it makes it less clear what that particular test is doing.

Summary

So what I hope I’ve highlighted that unit tests and integration tests are very different creatures and sometimes in the real world the edges between them become blurred. Unit tests do some thing better than integration tests and vice versa. You really can’t afford to rely on one without the other if you want confidence in the system your testing and when you understand those differences you can use them more effectively to target your testing against your code base