A few days ago I noted that Shigeo Shingo, one of the founders of lean manufacturing, once said that testing to find problems is waste, whereas testing to prevent problems is not. Today I’ve been helping out with configuring an instance of CruiseControl, one that runs three or four projects and checks that no-one has broken any of them. This is testing after the fact, after check-in, after the developer has written his unit tests and his code. So is it waste?
I think it’s quite a close call, but on balance I’ll differ with James Shore and answer “no”. There’s no getting away from the fact that CruiseControl is reviewing code after the fact. After commit. We have to break our project before CruiseControl helps us out. And I wonder whether the presence of a good, automated tester downstream can sometimes engender a little too much comfort. I know I’ve been guilty of saying “Oh let’s just check this in — CruiseControl will soon tell us if we’ve made a mistake.”
But in CruiseControl’s favour, it certainly does catch the most frequent problem I’ve seen with (developers’ use of) version control: forgetting to commit a new source file. And by running a complete, clean build, it can throw up environmental or portability problems not seen when developers work for long periods in stable, cosy workstation setups. So it’s a fresh pair of eyes, doing a quick review of the project’s latest state. Possibly akin to automated checks of a sub-assembly in a lean manufacturing plant, it catches some problems early in the flow.
Maybe in the end it’s about the team. Installing CruiseControl is a great first step on the road to being “always release ready”, and definitely helps teams transition to agile development. And then after a while, I wonder whether there’s a level of team maturity at which it may no longer be needed?
Sam W
September 12, 2008
> I know I’ve been guilty of saying “Oh let’s just check this in — CruiseControl will soon tell us if we’ve made a mistake.”
There was an air of astonishment among the team when I read this out. “When?”, they demanded in unison, flabbergasted, involuntarily spraying coffee across their monitors, “When has Kevin ever said this?” lol
Kevin Rutherford
September 12, 2008
Oh, I’m much older and wiser these days :D
Ashley Moran
September 13, 2008
I will also say no, but for a different reason. All the human investment in CI is done upstream, not downstream, so it’s still test-first. Is it waste to use autotest to continually run your test suite locally? I don’t think you could ever argue that – autotest notifies you of code breakage faster than you could ever verify yourself.
The biggest potential waste I could imagine is if breaking code is checked in, the entire team pulls the changes, and the entire team has to stop work to fix it. At least CI reduces the risk of that to pulls that happen during the build cycle (ie under 10 minutes).
But you can even mitigate that, assuming you are using a DVCS. The following would be trivial with darcs, as it can unrecord patches as if they never happened – I can’t say for git, Hg, bzr etc. Have the CI server maintain a repo/branch for every developer. When the developer pushes to it, have the CI server pull all new patches from the trunk (there should be none, as the developer should have done this locally!), and build from the developer repo. If it fails *unrecord the patch as if it never went to the CI server*. The developer can then ammend the patch and try again. On success only, push new patches to the trunk repo and build again. Basically – centralised, sandboxed autotest on a defined environment. You still potentially have a race condition where two patches are pushed (within 10 minutes of each other) that don’t break the build individually, but do combined; but if your communication is THAT dire, I suspect the CI server is way down your list of problems.
(Note that I’m not suggesting creating developer branches as a place to back up to on Friday afternoon, these staging repos would be fully automated, and either reject or integrate patches as soon as the new interim status is known.)