why YAGNI acts to EXPLOIT the bottleneck

Clarke asked me to explain my earlier throw-away remark that YAGNI forms part of the EXPLOIT step in moving the bottleneck away from development, so here goes…

YAGNI (You Aren’t Gonna Need It) is an exhortation from the early days of XP. It has been discussed and misunderstood a great deal, so I’m not going to get into the finesses of meaning here. For our purposes, it reminds the developer not to work on features or generalisations that may be needed, telling him instead to focus his present efforts on delivering only what he knows is the current requirement. (In the interests of brevity, I’ll refer below to YAGNI only in terms of added behaviour, and I’ll use the word “feature” for any fragment of any kind of behaviour; all other forms of YAGNI are assumed.)

(In my practice I use a similarly attention-grabbing soundbite. Whenever I see a developer do something “because it may be needed in the future” I accuse him of crystal ball gazing. I remind the whole team that it can be risky and dangerous to get your balls out, and that seems to help the message stick. Other times there’s an embarrassed silence.)

Writing crystal ball code has three effects: In the present moment, it means that the developer is spending current time investing in one of many possible futures; in the period from now until that possible future, it means that there is code in the system that doesn’t need to be there; and when the future arrives, it may look different than that which the developer predicted.

First, then, crystal ball code uses up current development time. This is bad when development is the bottleneck and when batch sizes are relatively small and when development order has been defined in terms of business value and when feature cycle time is a KPI. The time spent developing a crystal ball feature will delay the current batch and all batches upto the imagined future. There is a tiny chance that development of that future batch will be faster (see below), but all interim ROI (for example) will be reduced by the delay introduced right now.

Second, the crystal ball code represents inventory, and it has a carrying cost. This code, which may never be required by the end user, must always build, integrate and pass all tests; if ever it doesn’t, time must be spent fixing it. Furthermore, a larger codebase will always require more time and effort to understand and navigate (think of having to drive around piles of inventory in order to fetch anything or the lean practice of 5S). Even if the guess turns out to be correct, the additional carrying cost of this inventory will slow down the development of all batches of features between now and the imagined future.

Third, the developer’s guess may be just plain wrong. Either the imagined “requirement” is never requested, or it is requested and by that time the codebase is radically different from what it is now. The developer may have to spend time removing the feature (for instance if it would confuse or endanger the user) or completely re-design it to make it match how reality turned out. It is assumed that the “wow, that’s exactly what we needed” outcome is sufficiently unlikely that the costs of the other outcomes dominate.

So YAGNI is based on a few core assumptions:

  • The product is to be built incrementally in batches of features
  • Each increment should be potentially shippable in terms of quality and cohesiveness
  • It is hard to predict what features will be requested in later batches
  • It is hard to predict what future code may look like
  • Development is the bottleneck
  • Speed of development is crucial
  • The present value of current features is higher than the future value of future features

Under these conditions, YAGNI is part of the EXPLOIT step because it helps to maximise the amount of current development effort going into delivering current value.

counting the cost of broken windows

In Quantify the cost of the things you wouldn’t have had to do, if only… Dave Nicolette attempts to calculate the overall cost to his project of a single defect. He arrives at a total of $8,712.50! First, there’s the time spent by the latest developer who got caught by the bug; then there’s the time spent by his two teammates when they helped him debug and fix it; and then there’s the multiplier for all the times this same defect had occurred, but gone unfixed, during the previous two years.

Dave’s figures are compelling, and quite possibly fall on the conservative side – for example, there may have been occasions in the past in which the application hung up during testing, but no-one reported the problem.

I prefer to leave the figure in developer-hours (102.5 in this case), so that we have an indication of the amount of backlog that wasn’t addressed due to this bug: the broken window cost the project almost three developer weeks of progress! If that time had been used to add just one more story, or to release just a little sooner, I wonder what the knock-on costs of this bug would turn out to be, in terms of lost opportunity revenues etc…

Inventory in software development

One of the central tenets of lean (and TOC, for that matter) is that inventory is not an asset, but is waste. In particular, carrying inventory incurs storage costs.

American Bulk Warehouse Shopping

David Carlton is reading up on lean manufacturing, particularly in relation to applying its techniques to software development. This week David asks a deceptively simple question about inventory:

“… it’s not clear to me exactly what lesson [the carrying cost of inventory] has for software development”

While neither David nor I would claim that software development is similar to manufacturing, it can be edifying to map the themes, principles and practices of the one onto the other. I think the answer to this particular question lies in looking at the various kinds of inventory we have in software development, and examining the carrying costs of each in turn. Off the top of my head:

Unreachable code:
Still gets built, thus contributing to slower build times, and hence to reduction of flow and feedback during coding. It may also need to be read (or worse – understood) whenever we’re debugging or designing an extension in that area. We have to keep it compiling successfully, which means it could actually prevent design changes and cause inertia. And if it has tests, we have to keep them building successfully and passing – another potential cause of reduced speed, frustration and ultimately design inertia. (All of this because our codebase is part of gemba.)
Unused features:
Same as unreachable code, plus: Features we ship get in the way of our users, which will either confuse them, slow them down, or otherwise generally leave them feeling slightly uneasy about our product. And presumably our testers still have to ensure that the features do work as intended, so there’s extra pointless work to do for each release. And if these features have bugs, we have to carry the support cost when a user stumbles upon one.
Items in the product backlog:
Have to be reviewed whenever stories are prioritised. And if we have too many, they may contribute to a sense that the product is a Death March.
Artefacts that don’t ship:
By which I mean documents, plans and models that must be kept up-to-date as the fuzzy future comes into focus, and as the product evolves. None of these artefacts is directly a part of the product, and yet we spend time and money fiddling with them (while Rome burns).

(And we haven’t even mentioned the time and money invested in creating these things in the first place!)

What other inventory and carrying costs does your project have?

clean firetrucks

Seth Godin suggests that Clean Firetrucks are a sign that the firemen are focussing on the wrong things: instead of washing their vehicles they should be out preventing future fires. In my opinion – and who am I to decide how a fire station should be run? – they should be doing both.

If I were the manager of a fire station, I would be looking to clean firetrucks as a sign of many positive things – primarily that the lean notion of 5S was being applied. A clean, tidy, well-maintained workplace is essential to high throughput. Problems can be detected sooner, maintenance is easier and quicker, and morale is generally higher. Not to mention the firemen’s professional pride: who wants to be seen rescuing a puppy from a burning building, only to take it back to a vehicle that has ‘also available in red’ scrawled in the dust on the back?

In software development we have an additional workplace: our code. When we fail to keep it clean and tidy our productivity falls and our systems’ adaptability falls even faster. (This is where the exponential cost-of-change curve originates – untidy gemba.) When I see dirty firetrucks in a software shop, I know to expect low productivity, high turnaround times on feature requests, and defects in abundance. Broken windows, as the Pragmatic Programmers put it.

Lean thinking says that time spent keeping gemba tidy repays itself many times over in terms of increased throughput and sustainable pace. And besides, I happen to like clean firetrucks…

tidy gemba

My new manager wanted to give me some background reading to do today. So we went to the place on the fileserver where all the documents are stored, so that he could “show me around”. Naturally, we found loads of documents, but all of them were either obsolete or out of date. We never found the file he was looking for, and in the end I had to borrow a printed copy from someone’s desk.

We both wasted 15 minutes because the workplace (in this case, the filestore) was untidy. House-keeping is never a waste of time…

Update, 24.10.04
On reflection, this is probably a case of broken windows. One person leaves a folder untidy, and that gives implicit permission for everyone else to do the same. Because everyone can see that no-one cares.

a fish out of water

There’s loads to learn here, about what people do and about the way they do it. Fortunately many who’ve trodden this path before me have recorded their thoughts and findings; unfortunately, they wrote a jumble of documents that are strewn throughout a very disorganised filestore. I’m trying to do a better job as I record what I learn, but already I’m finding the lack of decent tools is slowing me down. This place needs a wiki!

everything in its place

A couple of weeks ago I finished reading Gemba Kaizen by Masaaki Imai. Just like all the other books on lean manufacturing, I found it extremely rewarding and thought-provoking, while at the same time being somewhat longer and more repetitive than I might have wished (the second half comprises an interminable list of case studies, many of which add very little to the value of the book).

The main idea I took away from the book was that of 5S. This is basically rigorous and institutionalised house-keeping (translated as Sort, Straighten, Scrub, Systematize, Standardize). The idea is that an untidy or disorganised workplace can contribute significantly to reducing the throughput of value for the customer.

Can this concept be applied to software development? I believe so. All our tools have to be readily accessible when needed. Our codebase, workspace and CM system should all be well-organised, and should contain nothing that isn’t needed to add value or promote flow. Our build should be clean and automated, and driven only from components that are CM-controlled. And when we’re working as a team, all our processes and techniques should be standardized so that each person’s actions are understandable by all.
Continue reading