the daily build

About four years ago, I was leading one of the teams in a 7-team software project. This time it was my turn to be the ‘integration manager’ for the next release. So very early in the release cycle I created a batch job that performed a nightly build of everything in the current codebase. And if anything had automated tests the batch job ran them too. I wanted to de-risk the release by finding most integration problems early, before ‘codefreeze’ at the end of the development phase.

The nightly build was a great success. Most nights the tests all passed, and only rarely did anyone check in code that broke the build itself. But the integration and system testing for this release was still a nightmare of panic fixes and hurried tests.

And a couple of releases later I worked out why: the existence of the nightly build had not changed anyone’s habits for the better. The other teams only ever checked in code on the last day of each release’s development phase, just in time for system testing! In fact I had made things worse than they were before: the fear of breaking the overnight build had actually caused many check-ins to be delayed until everything was “finished”. (And so our ClearCase repository crashed on the last day of every release’s development phase, due to overload.) Instead of providing feedback on development progress, the nightly build had actually reduced feedback and created more integration problems than we had before!

I’ve always advocated the daily build as one of the things that should be implemented first when moving a project towards agility. But I now realise that its success depends on a precondition: the development team (and its managers) must understand that it is possible to develop a system feature by feature. In the agile community we know all about the benefits of doing this, but I suggest that most software managers out there don’t even hold it as possible. Most of my peers on the project above thought it was impractical, or even laughable, to organise their work so that we had a working system at the end of each day. Why go to all the extra effort of finding pieces that could be added without breaking anything, and that were not dependent on something that was scheduled to be done sometime later…?

Update, 9 Sept 04
Mike Clark points out an interesting article in this month’s Better Software magazine on this very topic. There is a real difference between Continuous Integration and Daily Build, and the psychological impact of that difference will drive teams in quite different directions.

2 thoughts on “the daily build

  1. I’m wondering if it would have helped to add a trigger to mail the team on commits and/or generate a report showing the frequency of commits.

    Or better yet, use a modification-based continuous integration tool like CruiseControl. If not build passed notifications have showed up for a while, it’s pretty obvious no one is committing.

    This still may not be successful but you’ll at least be aware quite quickly what you’re up against.

  2. Hi Jason,
    When CruiseControl did become available, our local toolsmiths had already put a few man-months into “perfecting” the batch job, and now the whole distributed team was dependent on the dodges and hacks. So there was an enormous amount of resistance to introducing anything different, and we were hooked into a losing ‘tools vs. process’ war…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s