the toyota ballet

Over at Evolving Excellence today Kevin Meyer describes his experiences of a visit to Toyota Kyushu. Kevin is struck dumb by every aspect of the plant’s operation — for example this:

“Every movement is choreographed and improved. You have to stand and watch each person for several cycles to see just how choreographed. Each movement has a purpose and is designed to minimize movement for the current and next operation. As he is exiting the front seat after installing four bolts a slight twist of the left hand will open the rear door so as he swings out of the front seat he swings right into the rear. As he exits the rear he leaves a tool that is required by the next operator. This is very hard to describe, but it is truly a manufacturing ballet.”

Whenever anyone spots a problem they pull a cord to sound a bell:

“How often does the chime sound? About once a minute throughout the factory. There is no problem reporting a problem. The intent is to find it and get it fixed, fast. As in within a minute. How long does it take you to identify, report, and fix a problem? I know what you’re thinking, I’m embarassed too.”

To get the full sense of Kevin’s amazement, I recommend you read the whole article for yourself. The brilliant and breathless description makes me want to go to gemba and see for myself.

people problems vs process problems

dunce A good deal has been written recently on the origin of problems that occur during software development. Pascal started the ball rolling by noticing an apparent conflict between Weinberg and Toyota:

“… Gerald Weinberg tells us that Whatever the problem is, it’s always a people problem. […] The Toyota Way states that Whatever the problem is, it’s always a process problem.”

Pascal goes on to resolve the conflict by noting that it is a mirage: fix the “people problem” immediately, and then fix the “process problem” so that same class of problem can’t recur. Aside from dubious use of the term “people problem”, these are the classical elements of the jidoka method: detect, stop, fix, prevent.

Then Marc followed up by proposing that it’s a false dichotomy:

“Processes (by which I mean the things and actions that actually happen in a system) are the manifestation of what the people involved have learned – of both their explicit and tacit knowledge. People and processes are two sides of the same coin. The way to improve your processes is to improve your people.”

Dave, though, brings the question around to organizational culture:

“IMO the key question is, How does the organization — which comprises people (who have a culture) — react reflexively when things are going badly? Is the tendency to fall back on adherence to formal processes, or to facilitate people’s ability to find creative solutions?”

I’ve raised Pascal’s original dilemma a number of times recently during my jidoka workshops. At some point I’ll inevitably make a bald statement that I believe every problem is a process problem (and no problem is a “people problem”). But then I find the resulting discussion revolves around some participants’ interpretation of the word “process” as being equivalent to “procedure”. And my point is lost forever. So here I want to re-cast the debate, hopefully in terms that are unambiguous.

First, what do we mean by “problem”. I take the term to refer to any reduction of productivity in a system or organisation. In TOC or lean terms this usually translates to a loss of throughput or flow – the problem boils down essentially to one of the seven types of muda – but it can also apply to an increase in operating costs without a corresponding increase in throughput. And I think we’re using the word “problem” above to refer to two different things:

The bad things that happen and which trigger us to look hard at our system. TOC (the theory of constraints) calls these UDEs – Undesirable Effects. I prefer to call them incidents to reflect their uniqueness and their immediacy.
root causes
Those attributes of our system that lie at the heart of the observed incidents. Any given incident will have a single root cause, if we broaden the scope of our system far enough.

And by system I mean that collection of processes, procedures, situations, knowledge, structures that make up the environment in which the incident occurred. The system may be a single department and all its working, or it may be the broader organisation comprising the whole company.

So above, when I talked about “process problems”, I intended to refer to any root cause that is ultimately responsible for causing an incident. This will be some attribute of the system we’re operating within. It may be that the system didn’t encourage two people to talk to each other, or didn’t ensure adequate knowledge in its workers, or comprised an incorrect flow of materials, or failed to embody a quality check at some point, or whatever.

So, trying to be perfectly clear: I believe that the reductions in a system‘s productivity occur because of the structure and behaviours of that system. Randomly switching people around might cause different incidents to actually occur, but in every case it is the system that permitted those incidents, and the system’s attributes that ultimately caused them. It seems to me that looking for people problems is about as smart as blaming the laser printer for a badly spelled document. People are not the problem. Ever.

I leave the final word to the Lean FAQ of the Northwest Lean Networks:

“If your ‘5 Why’ exercise seems to be pointing to ‘operator error’ as the root cause, you are going down the wrong path. Operators only do what our production systems allow them to do, so the root cause is in our systems, not our workers.”

jidoka at SPA2006

After Norm Kerth’s keynote on Tuesday morning, I ran my Jidoka in Software Development session again. This time there were nine attendees, which I guess reflects the perception that this topic occupies a specialist niche. The discussion was lively and interesting again, and yet again we covered topics that had never come up before. (Dave Draper, one of the participants, has posted his reflections on the session here.) I never cease to be suprised and delighted by how the specific combination of individuals’ interests and experiences can produce new insights in a topic I had heard discussed a dozen times already. My thanks to all who took part, both in this session and its predecessors.

During the next few weeks I’m planning a series of posts detailing what I’ve learnt from these sessions about applying the lean jidoka concept to software development. In the meantime, several of you have asked me to describe the mechanics of the session itself. So for details, read on…

muda workshop 1 The session begins with the story of Paradise Mill, which I also relate to the early days of the Toyoda Spinning and Weaving company, where autonomation was first introduced. At this point it is important to point out that the metaphor of software development as manufacturing doesn’t work well. Developing software is more akin to designing the product, as opposed to making many copies of it. Consequently we need to ask the question: can a practice from lean manufacturing be translated to development at all?

Next we do the Walking About exercise. Participants pair up, and assign one member of the pair to be the “worker”, while the other is the “manager”. The worker must walk 60 paces in 1 minute, but may only do what the manager says. Meanwhile the manager may only say “go”, “stop”, “turn left” or “turn right”. In a crowded room with many participants and chairs and tables it turns out to be rare for anyone to achieve 60 paces. But then we re-run the exercise, this time with no managers, and with every participant able to direct himself. Of course, everyone completes 60 paces well within the time allowed. We then discuss briefly the difference in effectiveness between teams whose process and plan are imposed, versus those teams who are empowered to decide for themselves how to reach their goal. This empowerment and autonomy – together with a supportive incentivisation scheme – are a critical, albeit oft neglected, component of the success of lean manufacturing. (I’ll explore this more in future posts on the Jidoka workshops.)

Next I briefly run through a chalk-n-talk description of the classic elements of jidoka: detect the fault; stop everything; fix the problem; fix the cause of the problem. (I believe there’s a fifth element, whose importance is increased in the software development domain: measurement. More soon.)

At this point we move on to the centrepiece of the session: a simulation of a manufacturing line. The participants group into circles of around 10 people, with each person acting as a machine in a production line. (On reflection, having the groups stand in a line, instead of a circle, might have been clearer.) Each line will play Chinese Whispers, using as input a phrase supplied by me. The last person in each line writes the resulting “sentence” on a whiteboard at the front, and the results – usually very funny – are compared to the original. There ensues a brief discussion about late discovery of faults etc.

I then invite each team to re-design their process in order to achieve better results next time around. The only constraints are that the message must be passed to every team member by whispering. Most teams toy with pair-listening (these are agile programmers, after all), and with repeating back the sentence after hearing it. Many then settle on some variation of supplementing the whispering with a written version of the text. We then re-run the game, this time with much better – although rarely perfect – results. (If there’s loads of time available at this point, we may repeat this process, allowing the teams to explore ways to eradicate the small errors that crept in last time.)

Having seen the dramatic improvements in quality that can result from empowering a team to design its own process, we move on to mapping jidoka to software development. In teams or as a group (depending on the time available and the number of participants) we explore the four steps (detect, stop, fix, prevent), looking for things we already do under each heading. The takeaway from this session is that the ‘prevent’ step is what makes the approach economical, and I will continually bring the discussion back to that area so that we explore ways to change and evolve software development processes. Then we wrap.

(There’s a session handout too, which is essentially a dump of the SoftwareJidoka squidoo lens.)

paradise lost

In What creates job dissatisfaction? You’d be surprised Joe Ely relates the story of a recent job interview in which the candidate had been dissatisfied with her previous employer’s approach to quality:

… she was instructed, if she couldn’t fix the problem as the line moved past her station, to say something to the next person downstream, and then carry on with the next car. “We never knew if that problem got fixed or if it just ended up as the customer’s annoyance,” she told us with a frown.

Joe’s story reminded me of Paradise Mill in Macclesfield. Seems like so many folks have forgotten the wisdom of their forefathers.

use tests as a failsafe

Yesterday I wrote about tolerating the red bar, when a few scary tests fail only occasionally. And today it strikes me that one of the contributors to the persistence of this situation is our tool-set.

As I said, we run the tests, get a red bar (sometimes), and then open up the test list to check that only the expected tests have red dots against them. If we see nothing untoward we just get on with the next task. But of course we aren’t really looking at the reason for the failure. Perhaps the tool itself is making it too easy for us. Or perhaps we interpret those dots too liberally?

So here’s the thought: What would life be like if our tools actually prevented check-ins while there are any failing tests? This would effectively “stop the line” until the problem was sorted out. And it would force us to address each problem while that part of the codebase was fresh in our minds.

I also suspect that peer pressure (“hey, I can’t check anything in now!”) might quickly cause us develop a culture in which we tried to eradicate the root causes of test failures. Instead of relying on CruiseControl to “deodorise” our stinky practices…

(If you’ve tried this I’d love to hear your experiences. Drop me a line.)

jidoka at SPA2006

I’ll be running my trusty Jidoka workshop again soon, this time at SPA2006 in March. I ran the same session at XPday in London last November; see feedback from Lasse.

Lean manufacturing is based on two pillars: Pull and Jidoka. Agile methods focus most of their attention only on Pull: customers pull features or user stories from the development team, and the development team carries out every task just in time and without building up inventory. Jidoka – the policy of stopping the production line whenever a fault occurs, and then fixing both the fault and the cause of the fault – has been largely forgotten. Jidoka is what keeps value flowing fast through the process, iteration after iteration. And yet there are no published studies or collections of Jidoka practices as applied to software development.

The session’s objectives are to learn about Jidoka on a production line, and to discuss its application to adapting a software development process in a learning organization. Participants will play a Jidoka Game which demonstrates the beneficial effects of “stopping the line”. The group will then work together to describe, discuss and discover “jidoka moments” – points at which software development projects are, could be or should be stopped in order to prevent faults passing downstream. It’s a fun session, and I hope informative too.


I spent another enjoyable day at XPday5 yesterday. As always, it was great to meet up with old friends and colleagues, although I never seemed to have the time to get into real conversation with anyone!

I was there to run a new version of the jidoka session, this time including games and a working group. The new format was great fun, and I’ll be running it again at SPA2006 next spring.

My main highlight of the day was Tom Gilb‘s introduction to ‘quantifying any quality.’ This was a compressed introduction to the Evo method’s approach to requirements, in which projects set numeric goals for everything from usability to maintainability to innovation. I was stunned by this approach, and spent some time over lunch quizzing Tom further. Tom’s thesis is that most of what we call “stories” or “requirements” are actually solutions to deeper business and user needs. Tom ferrets out those underlying needs, quantifies them, and then leaves the development team to decide how best to meet them. This chimes well with some of my experiences, so I’ve vowed to study Evo in detail during the next few months.

reflections on xpdays benelux 2005

Last week I spent an enjoyable day at XPdays Benelux in Rotterdam. I ran a couple of sessions, attended a couple more, and met up with friends old and new. Here are brief recollections of the highlights in my day…

The day began with every presenter offering a 1-minute sales pitch for their session. I got to do two: one for Hexagonal Architecture and one for Jidoka. A minute is longer than you’d think, and I’d guess that most of the pitches were over inside twenty seconds. I decided to be a little different, so I had everyone stand on one leg – hopefully to demonstrate that the standard layered architecture model compromises agility…
Continue reading