Scalable TDD
One of the problems that we come across as our projects mature is that the number of tests increase and so do test suite times. It seems inevitable … something like the sun setting in the west and the CPU speed doubling every few years. The obvious problem with this inevitability is that it takes LONGER to test the entire application. Sweeping changes therefore take longer as the feedback loop increases andpairs have to begin context switching during the day … “Didit get to green yet? No …where were we?”
This introduces many other problems:
- It is very hard to check in every half hour if your test suite takes 45 minutes to run. Hmmm … you certainly can’t check in without testing first so you have to live with the longer checkin cycles and potentially more merge issues.
- Our tests typically use a local SQL Server database - the sameone thatthe tests runningin Visual Studio use when you run a test while developing. This means you can’t run adhoc tests as you develop while the test suite is running. This is a showstopper since it means the test suite time is dead for development. You also can’t browse the application locally since it is also using the database and chokes as data is changed underneath it.
- Your automated build time is now increasing (due to longer test times) so it takes longer to know if you broke the build and longer to get a build to deploy to a test environment.
Enough with the problems … how about some solutions? We have brainstormed on lots of different possibilities and have come up with a range of fixes:
- Run performance analysis on the test suite and looks for opportunities to decrease test time. We have begun this process using JetBrains dotTrace with some success. There is still more work to be done and it is hard tofind the timeas release dates approach. It is important to think of your test code as “production” and therefore it should be refactored and improved on a regular basis.
- Separate the database from the tests and look to use mock objects wherever possible. This is difficult for existing projects where sophisticated database logic is in use in several places and interfaces are not being used to allow easy introduction of mocks.
- Offload the testing to other machines in order to benefits from a parallel system with its own CPU and database server. This has been done with some success by using powerful workstations at remote locations when they are not being used.
Look for more solutions and discussion on this topic in the upcoming months.
How do you solve scalability issues on your TDD projects as they evolve beyond 6 or even 12 months of development?
Jonathan Cogley isthe CEO and founder of thycotic, a .NET consulting company and ISV in Washington DC. thycotic has just released Thycotic Secret Server which is a secure web-based solution to both “Where is my Hotmail password?” and “Who has the password for our domain name?”. Secret Server isthe leader in secret management and sharing within companies and teams.
Comments
2 Responses to “Scalable TDD”
Leave a Reply
We use CCNet and have multiple build servers that execute tests for different parts of our project. There is some duplication between the them, but it does reduce the build times.
Other than that we look for long running tests in the build report and try to optimize tests when possible, but I agree that this is a real issue that doesn’t go away.
We’ve been building some infrastructure which intends to address many of these failings. In particular we’ve built a server-side framework which integrates testing with the SCM. Developers check-in as usual, but the change is tested on a build/test farm before it is made visible to others on the team. Changes which do not meet the check-in requirements sit in a developer specific sand-box and don’t break others.
Using this approach we find we can check in quite frequently. Most changes pass the build/test gate and are quickly promoted. Others do not and we can go back and fix them. Either way, we can continue to get new work done because we aren’t waiting for unit tests on our desktop boxes.