silk and spinach

Several times today my online reading has brought me to Henrik Mårtensson’s Kallokain blog. Henrik has a great knack of taking concepts from lean manufacturing or the Theory of Constraints and creating worked examples to show how they play out in day-to-day software development.

Take for example what Henrik calls the DIP - design-in-process - the software development equivalent of inventory. DIP is any work that has been begun, but not yet shipped to end users.

An easy, and highly visible, way to measure DIP, is to make a Project Control Board. Use a whiteboard, or a large mylar sheet on a wall. Make a table with one column for each hand-off in your development process. [...] At the start of each iteration, write sticky notes for each story (use case, feature description, or whatever you use). [...] Whenever a developer begins to work on a new story, the developer moves the story from the stack of sticky notes to the first column in the table. The developer is then responsible for moving the story to a new column each time the story goes into a new phase of development. [...] The sticky notes in the table are your DIP. Track it closely. If you have a process problem, you will soon see how sticky notes pile up at one of your process stages.

This fits with everything I’ve written about task boards, but Henrik then goes on to show how to calculate the current value of your DIP - ie. the investment you have made in the unshipped work on the board. In his worked example, a requirements change causes 20 story points worth of DIP to be scrapped; and as each story point costs 1,000 EUR to develop, the requirements change means we must write off 20,000 EUR of development costs.

It is worth noting that some of the 1,000 EUR per story point is “carrying costs” for the work that was scrapped. Just as manufacturing inventory costs money to store and move around, unshipped software (not to mention uncoded designs) has to be regularly read, searched, built and tested. It was there all the time, and we had to respect it and live with it. So it slowed us down; we had to step around it, and we would have tripped along more lightly if it had never been there. Looked at another way: with lower DIP inventory our cost per story point would be lower; equivalently, we could achieve higher throughput for the same expense.

(Of course, there are two obvious reactions to this write-off cost: make changes really difficult, or reduce inventories and cycle times so that DIP is always lower. Each has its sphere of applicability, I guess…)

Earlier today I was reading about the key business measures used in the Theory of Constraints, and running some examples in my head while I tried to make sure I understood exactly what was being said. Here’s what I learned…

To recap, the only business measures considered by TOC are

Throughput (T)

Cash receipts for actual sales, minus the cost of the raw materials used towards those sales.

Operating Expense (OE)

The expenditure required to produce what we sold, including heat/light, payroll etc.

Investment (I)

The money invested up front, or tied up in the operation. This includes depreciation on machinery, buildings, and other assets and liabilities.

These can be combined to give other measures; for example return on investment:

ROI = (T - OE)/I

(For more detail see throughput accounting on Wikipedia, for example.)

What struck me were the units used to express these measures. The only one I’ve seen explicitly stated was Throughput, which is expressed as “cash receipts per unit of time” - let’s call it dollars-per-day. So in order for the ROI calculation to make arithmetical sense, operating expense (OE) must also be measured in dollars-per-day. So far so good.

Now I had been expecting Investment (I) to simply be a sum of money. But in that case ROI would turn out to have units of “per-day”, which is clearly nonsense. So Investment must also be measured in dollars-per-day, and now I see that makes sense too. And in turn, this all implies that ROI is just a number, a scalar value. (Better perhaps to express it as a ratio or a percentage - so that an ROI of 1.50 is easier to understand as a 3:2 return or a 150% return.) Other traditional measures are calculated as follows:

Net profit (NP) = T-OE
Return on investment (ROI) = NP/I
Productivity (P) = T/OE
Investment turns (IT) = T/I

If my logic above is correct, Net Profit is expressed in dollars-per-day and all the others turn out to be scalar, unit-less numbers - and that alone helps me understand them much more.

Making targets and their associated metrics more visible can be the most important factor in achieving those targets. That’s the message behind the agile notion of Information Radiators or Big Visible Charts: the visibility itself is a critical component of their success. Today Shawn Callahan tells an anecdote in which it seems the increased visibility of a target and its metric served to dramatically change employee behaviour. He relates a story from Pfeffer and Sutton in Hard Facts, Dangerous Half-Truths, and Total Nonsense in which a shipping company wished to save money by bundling small orders into bigger boxes. The policy wasn’t being followed, so:

“So the company announced a new program that provided rewards such as praise—not financial rewards—for improvement. On the first day, the proportion of packages placed in the larger containers increased to 95 percent in about 70 percent of the company’s offices. The speed of this overwhelming improvement suggests that a change in performance derived not just from the rewards that were offered, but also from the information provided that the current performance level was poor and this action—consolidating shipments—was important to the company.”

As Jack Vinson suggests, the relative invisibility of the target behaviour and its measures could well have been the constraint on that business at that time. The anecdote is from the 1970s, too early for the business to have used the ToC thinking tools to diagnose the situation, or for the metric’s increased visibility to have been the result of applying the five focusing steps (identify, exploit, subordinate, elevate, repeat).

For me, the lesson from this story is that the constraint is as likely to be in knowledge management or communication as anywhere…

A month ago I began asking whether it was possible, meaningful or sensible to measure the performance of an agile coach. Herewith a couple of further thoughts on that topic.

Firstly, there’s been some discussion of metrics over on the scrumdevelopment Yahoo group. The original question posed to the group was:

“What metrics do I collect that tell me this agile stuff is actually doing my group any good”

Mary Poppendieck weighed in with three metrics that should be taken together as a package:

1. cycle time;
2. business case realisation;
3. and net promoter score.

Her post goes on to describe these measures in detail, and why they are important. They do seem spot on to me - though I’ve only tried the first for real, and even then with only limited success. It seems to me that there’s a strong case for measuring the coach according to the results of the organisation being coached, and these look to be measures I’d be willing to adopt for myself. But then…

Secondly, I have now finished Value-based Fees: How to Charge and Get What You’re Worth by Alan Weiss, as recommended by Deb. While not about coaching per se, the book does make recommendations as to how a consultant should establish performance measures with his or her clients. Weiss divides measures into quantitative and qualitative. Regarding the former:

“[Y]ou cannot base your contribution on what I call a ‘magic number’. While a 6 percent sales increase might be highly desirable and even, in your opinion, very achievable, it’s never a good idea to peg your success to that magic number. The reason is that there are far too many variables that can affect that number adversely for you to take that risk. [...] It is better to state that you will assist in maximising the sales increase.”

And regarding qualitative measures:

“Qualitative measures can be among the most powerful [but] “I’ll know it when I see it” [...] is insufficient for a consulting project. In fact, here are the parameters for creating a successful, anecdotal sereies of measures for a project: the buyer himself or herself will judge the result; the effectiveness will rely on observed behaviour and factual evidence; there will be gradations of success, not success or failure; there must be reasonable time limits.”

Weiss goes on to give examples that are very much like the kind of anecdotal measures I’ve been using on my own projects. So now I’m back to where I started! On reflection, I think that’s just fine. Start with something like:

1. assist in minimising cycle time;
2. assist in maximising business case realisation;
3. and assist in maximising net promoter score

and then augment those with any other anecdotal measures that are valuable to the team, manager or organisation I’m working with.

What measures do you use to assess your own performance at work?

Since I opened this particular can of worms a couple of weeks ago, the power of intention-manifestation has brought these related blogs posts into view:

• In What is a coach worth? Deb Hartmann points out Value-based Fees: How to Charge and Get What You’re Worth by Alan Weiss. My copy arrived yesterday, and I’ll report back here when I’ve read it. The table of contents suggests that it might be useful.
• In How do you measure up? Dwayne Melancon points out the book Measure of a Leader by Daniels and Daniels. Not directly related to measuring the value of a coach, but not a million miles away either. I wonder what actual measures they propose.
• In Agile Atlanta - 4/11 meeting notes John Brothers reports on a discussion of metrics for agile projects. Again, not directly related to coaching, but it always helps to be able to measure team effectiveness along the way.

Do you know of any more discussion, research or case studies into measuring the performance of managers, coaches or businesses?

Software process improvement is all very well, but how do we know whether it is working?

The topic of measuring the effectiveness of process improvement activities has crossed my path several times in recent days. For example my current client and I are trying to define ways to measure the benefits to their organisation of retaining me. And then there was the designing metrics session at SPA2006; in their handout notes, Jason and Duncan suggest that process improvement be run as an agile project, with its own stories, velocity etc. Not to mention a conversation I had with Deb Hartmann on the same topic; seems she wants to measure - or at least chart - her coaching activities.

I’ve found almost nothing written about this topic, so at present I’m afraid I have only questions. First though, some basic assumptions. I assume that software process improvement in itself is pointless. As I said earlier this week, the term “process” doesn’t capture enough of the behaviour of a system to be useful. And of course “improvement” is a relative term. Improvement compared to what? And towards which goal(s)? According to Goldratt there’s only one Goal: maximising profits in both the present and the future. I will assume, then, that software development is always carried out with that goal in mind: the software either reduces our own organisation’s costs or increases our sales, or is designed to be sold to other organisations so that they can use it for one of those ends. Either way, the process of producing that software is only a fraction of the whole system, and may contribute a large or small amount to how well we’re working towards our organisation’s Goal. Furthermore, improving the software development department in isolation may prove beneficial initially, but could be creating a local optimum at the expense of another part of the system, or could simply not be worth doing, if other parts of the system are contributing more to poor overall performance.

All of that notwithstanding, there will be times when an organisation must turn its attention to improving the software development department. Software development will inevitably be the system’s primary bottleneck some of the time, and so its contribution to the goal will need to be addressed. But depending on context the “improvement” required may involve improving quality, reducing time to market, improving customer service, ensuring the right software is developed, training people, or any number of other things. That is, the day-to-day dynamics of the overall system’s push towards the Goal will create short- and medium-term demands on software development; and the response from software development must be measured in terms of those demands.

Henceforward, I’m going to use the woolly term effectiveness as a shorthand for the software department’s contribution to the overall system Goal. Improving the department’s effectiveness then equates to reducing costs and/or increasing sales in the overall system. And instead of talking about “software process improvement”, I’ll discuss “software department effectiveness” instead.

So, back to the top. How can we measure the performance of a software coach? As yet I have no answer; instead I’ll ask a few more related questions:

1. If we hire a coach or process facilitator or scrum master, how will we know whether we’re getting value for money? Unless the coach doesn’t charge, the organisation’s Operating Expense has increased, so should we be looking for a corresponding increase in sales?
2. Should a software coach charge a percentage of their client’s increase in profits? Now and in the future?
3. How often is it possible to attribute improved profitability to the actions of a software coach?
4. What if the service constitutes training - how can that be directly accounted to profitability?
5. Given that increased profitability may be measuable only after a period of time has elapsed, how should the risk be apportioned (between client and coach) during that interval?
6. Conversely, how risky is it to define local measures purely on the software department?
7. Imagine an ineffective organisation whose biggest problem is not within the software development sub-system. But suppose nevertheless that the organisation blindly hires an agile coach to improve the software department’s effectiveness. Is it meaningful to talk about “improvements” when the real problem is elsewhere? Can the changes be measured in any meaningful way?
8. Assume that one of the objectives of a coach is to transfer sufficient knowledge so that the client organisation can ultimately continue alone. How can we chart progress towards that goal?

I can find no answers in the lean manufacturing literature (assuming lean sensei charge for their services), or in the agile domain. I suspect that we as an industry just don’t know (yet). Deb suggests we may need a few OpenSpace discussions to work out the answers - or even to establish some direction - and I concur. If you know of any resources, or if you wish to participate in getting these discussions going, please drop me a line.

The second session I attended at SPA2006 was Designing metrics for agile process improvement, run by Jason Gorman and Duncan Pierce. Their thesis is that metrics are difficult to design, and making a mistake can lead an organisation down the wrong path entirely.

To demonstrate this, they divided us into four groups and assigned us each a “goal”. My group’s goal was “more frequent releases“. We had to brainstorm metrics that might act as measures of an organisation’s progress towards that goal, and then pick one to carry forward to the remainder of the exercise. We spent some time thrashing around wondering why an organisation might have such a goal, and finally we plumped for the metric average time in days between releases.

At this point, one person in each group took his group’s metric to the next table, where that group (minus one) would attempt to “game” it. The objective of this part of the afternoon was to try to get the metric looking right, but such that the organisation was clearly doing something very undesirable behind the scenes. Our table was joined by Sid, bringing along a metric whose stated goal was “fewer bugs in the product” and whose metric was number of bugs per function point per release. During the next half-hour we had great fun trying to find ways to write more bugs, while still having fewer per function point. These included testing previous release really hard and fill up the bugs database, then hardly test future releases at all; and making it really difficult for users to report bugs. Meanwhile another group was gaming our own metric, by producing zero-change releases and by saving up releases into rapid-fire batches all at once.

Next, if I recall, we had to re-design our original metrics in the light of the panning they had received in the hands of the gamers. We ended up measuring the goal of “more frequent releases” using the metric days per story point between customer feature request and customer acceptance. We tried to phrase it so that it averaged out over easy and difficult features, and so that the count forced us to release the feature to an end user. This neatly side-steps all attempts at gaming (that we could discover), and is roughly equivalent to throughput - or to Kent Beck’s software in process metric.

For the workshop’s final segment, we recognised that a single metric by itself is rarely sufficient to direct process improvement. Change one thing for the better and something else may well slip out of line. Jason and Duncan’s thesis is that we need a system of metrics, each of which balances the others in some way. So as a final exercise, we began an attempt to take the metrics of all four groups and weave them into a system, showing which measures reinforced which others etc. For me this is the key observation from the whole afternoon, and I would like to have spent more time here. I had never really seen the attraction of Systems Thinking, but these models derived from the need for metrics do seem to be a good application of the technique. Food for thought indeed…

Here’s an absolutely mad thought-experiment…

Over at KnowledgeJolt, Jack Vinson is on a “personal effectiveness kick”. And being interested in the Theory of Constraints, he’s naturally looking to define his Goal, and to define one or more effective measures of progress towards it. In response to a little goading from me, he says this in the comments on Too Busy Being Unproductive to Learn to Be Productive:

“What we need to know (in terms of measures) is whether we are reaching our goals or not. If we’ve taken action that moves us in the wrong direction, that’s probably the wrong action. (If we didn’t know that it would be wrong in advance, then it was a good learning experience, and we can do something different next time.)

“The difficulty with productivity, as it is typically used, is that it simply measures output without looking at direction. For example: I did ten things, but only one of them contributed to my long-term goals. Even worse, two of them distracted from my ability to meet my goals.

“I’m not sure what the final personal measure would look like, however. I’d rather have something that showed how much progress I’ve made, rather than an accounting of how many things I’ve done.”

What’s required, then, is a way to define our goals - and to quantify them.
Read the rest of this entry »

In The Productivity Metric James Shore adds fresh fuel to the “running tested features” debate. The article is well worth reading, and James concludes:

“There is one way to define output for a programming team that does work. And that’s to look at the impact of the team’s software on the business. You can measure revenue, return on investment, or some other number that reflects business value.”

I whole-heartedly agree with this conclusion, although in my experience there are a couple of hurdles to overcome:

First, the figures may be hard to come by or difficult to compute. This is particularly true of infrastructure software, or tooling that’s used internally to the business. How do you compute the development team’s ROI from the impact they have on an admin clerk’s day-to-day? There will always be the danger of monetising some intermediate measure, and thereby creating a local optimum. (If you have examples of this being solved successfully, please get in touch.)

And second, the development team may feel that the figures are too dependent on other business processes, such as sales or dispatch. Even where the software is the company’s product, the value stream is often not as short or highly tuned as one might wish; and so the developers may not wish to be measured against the whole stream’s effectiveness. In theory, rapid feature development and compelling usability ought to energise the sales team and the market to the point where demand dominates supply; in which case the value/time metric will work well. In practice, the necessary pull is too often absent. (Maybe in that case the metric is still valuable, telling us something about the whole value stream…)

One of the projects I work with has very high churn in the backlog. There are repeated and frequent seismic shifts in the project’s priorities, and each time that happens a whole slew of twenty or more cards instantly appears at the top of the pile. And a couple of times the entire backlog has been junked - sorry, archived. Naturally this all has side-effects, some good and some not so good.

It’s important to note that this project is considered a roaring success. The team has achieved amazing things in a short time. And its ability to cope with the backlog churn - while maintaining velocity - is remarkable. And yet the churn is disorienting: the team’s retrospectives regularly discover that individuals feel the project is out of control, or directionless.
Read the rest of this entry »