It seems to me that one of the principal reasons for the apparent rapid rise of agile methods in software development is the romanticized battle between programmers and managers. Spend any time reading any extreme programming discussion group, for example, and eventually the topic of adopting XP in the face of management resistance will crop up. Agile methods, and XP in particular, would appear to be the saviour of the oppressed programmer; finally the evil tyranny of the waterfall and the heavyweight methodology are crumbling in the face of guerilla resistance from programmers who knew they were right all along! Or that’s how it seems to them.
Does the extreme (sic) zealotry of the programmers represent years of pent-up frustration with the situation in which they found themselves? Do they blame management for controlling them, for keeping them strait-jacketed in heavyweight methods for so many years? If that’s the reason, why were such methods of control deemed necessary in the first place? Two words: manufacturing metaphor.
The naive reading of the manufacturing metaphor requires that programmers are the unskilled labour implementing the designs of the talented few, and managed by superior beings who are somehow more mature because they know about risk, business priorities etc. As time goes on I’ll blog my thoughts about more subtle and complex ways in which this metaphor can have real value for software development. But for now I’ll simply note that it was the most convenient and successful model to fall back on when the world called upon the software industry to grow up. Heavyweight methodologies are a natural elaboration of the metaphor, and provide a way to control the programmers by imposing some variant of the manufacturing metaphor.
Unfortunately the apparently necessary chains of command required by the metaphor can also be easily hi-jacked by egos. Instead of being purely a facet of the methodology, they often become self-perpetuating hierarchies. Further ‘process improvement’ is then too often motivated by fear of falling. By treating talented people as machines, while simultaneously removing many of their powers and choices, management hierarchies can remove one perceived source of risk – programmers thinking for themselves. This leaves the programmers feeling controlled and not trusted, but satisfies the manager’s need for control in an uncertain world.
So it seems to me that therein lies the origin of the XP zealotry. The use of an inappropriate metaphor for software development has created, on the one hand, programmers’ resentment at being treated like machines or talentless drones; and on the other, the needs of a whole generation of IT managers to control what they feel they cannot trust. I’ve been in both camps at various times, and one of the hardest things I ever had to do as a manager was to let go, to stop trying to control everything and trust that intelligent people working as a team need only to be facilitated. The results have been very rewarding, and I would never (knowingly) go back to managing by control. I certainly hope that the agile manifesto‘s focus on people has a broad impact on software development in general, long after the hype and divisive arguments have quietened down.
Update, 13 Sept 04
What could replace hierarchies? Perhaps if the egos weren’t in the way we could try roles and collaborations?