Sounds great, but what is the cost?

My latest posts have been very developer-centric. This post should address a broader audience, including those with suits and ties. Yes, managers and CEOs, I am talking to you! ;-)

Test-driven development is not only about writing code, however. Software development is about delivering value to a customer and earning money. Profit is income minus costs. One of the most common arguments against TDD is "it takes too long time" or "it is too expensive". So what is the cost of doing test-driven development?

It's not easy to put a price tag on test-driven development. We can get an idea about the alternative (not doing TDD) by looking on the Total Cost to Market, however. This is the total cost of a change in an application from idea to customer deployment. The total cost includes

• Developing the feature
• Testing
• Bug fixing
• Cost of risk that there are defects
• Customer deployment

The cost incurred by actually typing the code is just a fraction of the Total Cost to Market.

Total cost to market

How much does it cost to develop a given feature?

In the beginning of a development phase, this is a fairly simple calculation: it's the number of hours spent on coding multiplied by the developers' hourly rate.

As time progresses and the product grows, this price increases because we add the risk of introducing bugs that are not discovered. A bug which is introduced late in the development cycle is more costly because there is an added risk that this bug (or a new bug that is introduced by the bug fix) is never discovered.

It becomes even worse after feature freeze when beta testing has started. If a bug is discovered after 90% of the testing has finished, how do we know that the bug fix did not introduce a new bug affecting the previously tested features? Do we test everything over again? Now that's an additional cost that contributes to the exponential cost graph!

And now the nightmare begins: what if the bug is discovered by the customer instead of our testers? That adds the additional cost of lost customer satisfaction and re-deployment. We can't easily put a number on this which can be put into a budget, but we certainly want to avoid that!

Fail early

The moral is; we want to fail early. We want to discover bugs as early as possible in the development cycle. The developer should get an alarm bell as soon as he creates the defect and not when a beta tester (or even worse, the customer) discovers it.

Bugs will occur, so we want to find and fix them while the cost is low. In other words, we want the bug to contribute as little as possible to the Total Cost to Market.

Good unit tests is a good first line of defense against bugs. Surely it's not a guarantee that you will deliver bug-free software, but it's a very efficient tool to deliver software with fewer bugs. Little research is done to actually quantify the economical benefit of TDD, but this study done on a few projects in IBM and Microsoft indicates that TDD roughly gave 25% increased developer time and reduced the number of bugs to a third: http://research.microsoft.com/en-us/groups/ese/fp17288-bhat.pdf

Conclusion

Did I answer my own question about the cost of doing TDD? No, I can't provide you with the number of dollars to put into your budget spreadsheet. However, it's pretty obvious that we can reduce the Total Cost to Market by discovering and fixing bugs as early as possible. That is a task which is far too important to rely only on manual testing.