Preface

Some years ago, I was the solution architect on a brownfield project. A new strategically important product was to be built, which required a large rewrite of the old existing monolithic application, and, consequently, the surrounding services had to be modified. Not only because of these new additions, but mostly due to adjustments to the core and its APIs. A massive undertaking that was too complex for a single person to keep in their head. Especially me.

I had been brought in for my experience, and it was exactly that experience that told me clearly that designing this thing alone was a fool’s errand. So I did something I had learned from the Domain-Driven Design (DDD) community; I told the team we were going to design it together. I had no intention of being the master designer; all architecturally relevant decisions should be done collaboratively. I was one of them, with the perspective of someone who had seen a few of these complex undertakings before, but they were the experts on this product, this codebase, and this domain. We had to model together, decide together, and own the design together. I was not equipped to do this alone, and I knew, without telling them, that they would probably never accept my design regardless of how perfect it would be. Which it would not.

It worked, and not in a small way. It worked the way the Agile Manifesto promises but rarely delivers. The team came alive. The architecture they shaped collectively was better than anything I could have dreamt of drawing on my own. The line management saw it, the project lead saw it, and the fellow coders in the department were impressed. I felt I really had made a proper change for once, found a way out of the frustration of being a foreigner in their system and being expected to perform miracles. Lifted tons off my shoulders. And brought excitement to us all. My imposter syndrome was nowhere to be seen, even.

Then the estimates began to slip, and we started missing expected deadlines. We had been clear from the start that the estimates were pure guesses, as the rewrite was something none of us had done before, and the monolith had so much technical debt and history that was older than all of us. The system landscape was complex enough that no one could really know how long this would take, and we had said so, in writing, more than once. But complexity is uncomfortable for organisations that need to plan, and a few months in, the planning anxiety started to outweigh the evidence. A new layer of management arrived as well, who had not been part of the work up until then and did not know the team or the systems. They looked at the slipping numbers, looked at the architect with the title and the grey beard, and made the decision they thought was responsible. They asked me to take control.

I tried to explain that the team had earned the design, that taking control from them would dismantle the thing that was making the work possible. It did not land, and the pressure mounted. The trust the team and I had built collapsed in a matter of weeks, replaced by the older and more familiar arrangement where one person directs and the others execute. The team even accepted it as they were all too used to the idea that this is how things really work, and that what we had was fragile in ways none of us had named.

I left not long after. I was disillusioned in a way I had not been before, and I did not yet have the language nor the understanding of what had happened. I think I know now and would like to share it with you all. Hoping that it can help you avoid the hit that we all took back then.

What I had seen was real; the work had been good, and the collapse had unfortunately been equally real. Something had been working, and then something else had broken it, and I could feel the shape of both even though I could not yet name them. So I began to read. A lot.

My route in was systems thinking. I had been drawn to it for a while, partly through the DDD community and a colleague who could not stop talking about Thinking in Systems by Donella Meadows. I had also been an avid follower of Ruth Malan’s work on architecture and scopes, where there are different levels of architectural decisions to be made, ranging from the details in the code to large-scale enterprise-level decisions. That, and some really inspirational recordings of Russell Ackoff on YouTube, gave me a vocabulary for why some messy problems do not yield to being broken into neat little parts. Some of the material also kept pointing back to something older and stranger, something called sociotechnical systems design. I followed the thread, that rabbit hole.

And that was a deep one. It led me to the Tavistock Institute of Human Relations and to the British coal mines in the 1950s, where researchers had found that workers organising themselves into self-managing groups produced better results than the supervised division of labour the new technology had imposed. It led to a chapter of Norwegian industrial history I knew, embarrassingly enough, nothing about. In the 1960s, as part of the Industrial Democracy Programme, Fred Emery and Einar Thorsrud ran field experiments across four companies on whether workplaces could be made genuinely democratic. It then led me to Fred Emery’s later work, the Open Systems Theory, which I would come to rely heavily on, where the design of organisations is treated as a question to be answered by the people who do the work, rather than by external experts. Felt emotionally familiar.

What struck me most was a single insight that ran through all of it. It is only when people design their own work that they develop the motivation, responsibility, and commitment to implement it well. Fred and his partner Merrelyn Emery had written this back in 1974,¹ and by pure coincidence, I had stumbled onto a version of it through collaborative modelling in the DDD community half a century later. The miners had lived it in the 40s and found the same solution as we did in that monolith rewrite. I had assumed I was doing something new on that brownfield project. I was doing something very old and deeply human, it turned out, but unfortunately, with most of the supporting structure missing. Which led to its unavoidable demise.

The next thing that struck me was the language of organisational design principles. Emery had named two basic structural patterns on which organisations can be built. The first is where coordination and control live above the work, in supervisors, managers, and architects. The second is where coordination and control live with the group doing the work. He had shown, with research, even in my Norway, that these two cannot be mixed.

The attempt to mix them actually resulted in a mixed mode with no clear home for coordination and control. The Emerys clearly showed that this adds confusion to the mix, as nobody really knows who calls the shots, and it results in what they called laissez-faire, which performs worse than either of the other two. Even worse than the slow and rigid bureaucracy. That was the piece that made everything else legible. The brownfield project had been an island of the second principle inside an organisation built on the first. The collapse had been the first principle reasserting itself under pressure, which is what it does when its assumptions are challenged. The vocabulary made what I had lived through legible.

I have spent the years since trying to bring this thinking into the IT industry. The research behind it spans sixty years and dozens of industries: coal mines, chemical plants, shoe factories, oil refineries, wineries, hospitals. The results are consistent and well-documented. The IT industry is conspicuously absent from that list. This book is a modest attempt to change that. The industry has its own version of the same story. It has been trying to move toward something like self-management for over two decades under the banner of Agile, and has produced mostly laissez-faire instead. The research is precise on this,² and the experience confirms it. Most of us in the industry have lived some version of what I lived through. Most of us do not have the language for it.

This book is an attempt to give us that language and show you where and when to recognise it.

A word on language. The entries here name things the way people experience them, not the way the theory describes them, and that gap is intentional. This is practitioner writing, not a contribution to OST science. The precision sacrificed for recognisability can be recovered by following the references — and if an entry sends you to the primary sources, that is exactly the right response.

The examples in this book are from software organisations because that is where I have spent my working life. But the structural patterns are not industry-specific. Wherever people come together with a shared purpose, whether at work, in a community, an NGO, a volunteer project, or a neighbourhood committee, the same design principles apply, and the same dysfunctions follow from the same structural causes. The coal miners knew this. The Norwegian factory workers knew this. The IT industry is simply the latest to rediscover it.

This book grew out of a daily LinkedIn series. The format it found there, one dysfunction at a time, named the way people experience it, turned out to be the right one.

This book is therefore mainly a catalogue of organisational dysfunctions, each one named the way people experience it rather than the way theory describes it. Power asymmetry, not communication problems. Powerless retrospective, not ignorant developers. The incapacitated middle managers, not an agile transition in progress. The unrelatable mission statement, not people resisting alignment. Each entry sets up a situation that anyone who has worked in a software organisation will recognise immediately, then explains it through the lens of Open Systems Theory.

The aim is twofold. The first is recognition. If you have been carrying a dysfunction around without language for it, this book may help you name what you are inside of. Naming is not the same as fixing, but it is the precondition for fixing. You cannot redesign a system you cannot describe.

The second aim is structural. Most of the dysfunctions in this book are usually explained as failures of culture, leadership, communication, or individual character. Almost none of them are. They are structural outputs of a particular way of organising work, the bureaucratic hierarchy that has been the default for most of the industrial era and most of the IT industry within it. Once you see this, you stop trying to fix the people inside the structure and start asking different questions about the structure itself. We need to get out of this bureaucratic hangover we seem to be still in, Agile or not.

This book does not, however, give you a method for changing your structure. That will be the work of a different book, which I am writing with expert facilitator and organisational designer João Rosa, called Intentional Organizations, which will go deeper into the practice of structural change. What I do here is name what is broken, with enough precision that the work of redesign becomes thinkable. Diagnosis before treatment. Most organisations skip the first step and then wonder why the second step does not take.

A note on how to read this. The book is organised into five chapters that move from the most intimate dysfunctions to the most systemic. Chapter one is about workshops and meetings, where dysfunction is most visible in the moment. Chapter two is about teams; Chapter three is about departments and the middle layers of organisations; Chapter four is about the organisation as a whole; and Chapter five steps outside and looks at the organisation in its environment, where Open Systems Theory has its sharpest things to say.

There is also an intermezzo, midway through, where I take on the objections that anyone serious about this work eventually hears. That self-management does not scale; teams must have leaders; it will never work for knowledge work like IT; and why bother, Agile works fine. None of these survives contact with the evidence, but they are honest objections and deserve honest answers.

You can read the book straight through, which I recommend if you are new to the framework. You can also dip in. Each entry is built to stand alone, and the section introductions repeat enough of the vocabulary that you can land anywhere without losing your footing. The book is also designed to be read slowly. One entry a day is the original cadence of the series, and I still think it is the right cadence. Each diagnosis takes a moment to sink in. There is no rush. Download it to your favourite e-reader and read one entry on the commute to the office, and then compare it to what you observe there.

There is a third way. The part introductions, the intermezzo, and the two closing chapters together form a coherent, standalone introduction to Open Systems Theory, readable in an evening without visiting any of the individual entries. If you want the framework before the catalogue, or if you come back to the book later and want a refresher without the full catalogue, that skeleton works on its own.

A note on versions. This book will evolve as good digital books ought to. The LinkedIn series on linkedin.com/in/trondhjort continues to publish new entries, and each new release of the book incorporates them. The version you are reading is v1.0: a complete and coherent argument, but not the final word. Later releases will add entries to every chapter and deepen the intermezzo. If you are reading this on Leanpub, you will be notified when a new version is available.

A final word before the book begins. The dysfunctions in this catalogue are real. I know. They are also painful, particularly for people who have spent years inside them without knowing what they were inside of. If you find yourself recognising too much, that is not a sign that something is wrong with you. It is a sign that the structure you are working in is producing what it is designed to produce. Most of us have been there. Most of us are still there. The first step out is being able to name where you are.