Until recently I had always thought of CruiseControl as a backstop, something that would check for silly mistakes during commit, or certain classes of environmental dependency in build, tests or deployment. And then, during December, two things happened to change that view; and now I use CruiseControl quite differently.
As you know, I’ve been developing this tool called reek, which looks for code smells in Ruby source files and prints warnings about what it finds. reek has been developed largely test-first, and now has a couple of hundred tests. The first thing that happened was that I began to be irritated by the huge runtime of the full test suite — 15 seconds is a long time when you’re sitting waiting for feedback, and I noticed I would sometimes dip off into the browser while waiting for green (or red). I needed the coverage, but without the waiting around.
The second thing that happened was that during one of those interminable 15-second waits I read about the Saff squeeze. Then I tried it, enjoyed it, and found that the tests it produced were significantly different from those I wrote “myself”.
So now I use CruiseControl differently. I began by splitting the tests into two suites: fast and slow. The fast tests comprise over 90% of the total, and yet run in under 1 second; I use them as normal during my development cycle — no more waiting around! The slow tests consist of runs of the whole tool on known source code; there are only a dozen of them, but they take 14 seconds to run. (You might call them “unit tests” and “integration tests”, and indeed that’s what the two suites comprise right now. But the key attribute is their speed, not their scope.)
So now the slow tests only get run by CruiseControl, not by me (it does run the fast tests too, btw). I’ve delegated the 14 seconds of waiting, and now my development state of flow is much easier to maintain. And if the slow tests fail, it means I have one or more missing fast tests — so I whip out the Saff squeeze to go from one to the other. I’m using CruiseControl as a batch test runner, and the Saff squeeze as a mechanical tool to help generate a missing fast test from a failing slow test. (The process of squeezing also tends to tell me something quite interesting about my code. More on that another time.)
I’m sure this use of CruiseControl isn’t new or original, but I’m pleased with how well it works. Let me know if you try something similar?