the daily build

About four years ago, I was leading one of the teams in a 7-team software project. This time it was my turn to be the ‘integration manager’ for the next release. So very early in the release cycle I created a batch job that performed a nightly build of everything in the current codebase. And if anything had automated tests the batch job ran them too. I wanted to de-risk the release by finding most integration problems early, before ‘codefreeze’ at the end of the development phase.

The nightly build was a great success. Most nights the tests all passed, and only rarely did anyone check in code that broke the build itself. But the integration and system testing for this release was still a nightmare of panic fixes and hurried tests.

And a couple of releases later I worked out why: the existence of the nightly build had not changed anyone’s habits for the better. The other teams only ever checked in code on the last day of each release’s development phase, just in time for system testing! In fact I had made things worse than they were before: the fear of breaking the overnight build had actually caused many check-ins to be delayed until everything was “finished”. (And so our ClearCase repository crashed on the last day of every release’s development phase, due to overload.) Instead of providing feedback on development progress, the nightly build had actually reduced feedback and created more integration problems than we had before!

I’ve always advocated the daily build as one of the things that should be implemented first when moving a project towards agility. But I now realise that its success depends on a precondition: the development team (and its managers) must understand that it is possible to develop a system feature by feature. In the agile community we know all about the benefits of doing this, but I suggest that most software managers out there don’t even hold it as possible. Most of my peers on the project above thought it was impractical, or even laughable, to organise their work so that we had a working system at the end of each day. Why go to all the extra effort of finding pieces that could be added without breaking anything, and that were not dependent on something that was scheduled to be done sometime later…?

Update, 9 Sept 04
Mike Clark points out an interesting article in this month’s Better Software magazine on this very topic. There is a real difference between Continuous Integration and Daily Build, and the psychological impact of that difference will drive teams in quite different directions.