Leanpub Header

Skip to main content
The Leanpub Podcast Cover Art

The Leanpub Podcast

General Interest Interviews With Book Authors, Hosted By Leanpub Co-Founder Len Epp

Listen

Or find us on Stitcher, Player FM, TuneIn, CastBox, and Podbay.

The Leanpub Podcast 🎙 Feat. Jules May, Author of Extreme Reliability

Episode: #341Runtime: 1:16:46

The book by Jules May on Leanpub

Jules May - Jules May, author of Extreme Reliability: Programming like your life depends on it, joins Len Epp on the Leanpub Podcast to discuss why software fails, what safety-critical aviation systems teach us about writing bug-free code, and why the software industry lacks a cultural canon. Jules also shares his writing process and why he chose Leanpub over traditional publishers.


Jules May on Extreme Reliability and Writing Software That Never Fails

In this episode of the Leanpub Podcast, Len Epp interviews Jules May, freelance consultant and author of Extreme Reliability: Programming like your life depends on it. Jules specialises in safety-critical systems, mathematical software, and compilers, and has spent nearly fifty years in the industry. The conversation ranges from the early days of aviation software to the philosophical underpinnings of reliable code, and wraps up with a candid discussion of Jules’s writing process and his decision to publish on Leanpub.

From Aviation to a Career in Reliability

Jules began his career in the early 1980s working on software systems for aircraft, at a time when the industry was attempting to apply formal mathematical proofs to both hardware and software. He explains that although full proof of correctness turned out to be impractical for the complexity of real systems, the discipline instilled in him a deep commitment to reliability — one he has since carried into medical instrumentation, automotive systems, and enterprise software.

Why Planes Don’t Crash Because of Software

One of the most striking claims in Jules’s book is that no passenger has ever been killed by a software fault in aviation. He addresses the widely misunderstood Boeing 737 MAX disasters in detail, explaining that the MCAS stall-prevention software was doing exactly what it was designed to do. The crashes resulted from a combination of deferred maintenance on faulty angle-of-attack sensors, inadequate pilot training, a failure by Boeing to inform crews that the new system existed, and — in at least one case — cockpit communication culture that prevented a co-pilot from overriding a captain’s incorrect response. The software, Jules argues, was not the culprit.

Why Software Is So Brittle

Jules argues that mainstream software development actively engineers fragility into systems. Error-handling patterns — from legacy error codes to modern exception throwing — cause programs to abandon their execution context the moment something goes wrong, amplifying small failures into catastrophic ones. He contrasts this with well-designed physical systems such as aircraft and boats, which are built to be intrinsically stable: they can absorb disturbances and return to safe behaviour without human intervention. The goal, he says, is not perfection but stability — software that keeps working even when the environment misbehaves.

Reliability Techniques in Practice

As a concrete example of a reliability technique, Jules explains the Null Object pattern. Rather than scattering null checks throughout a codebase — which programmers routinely forget — a null object exposes the full interface of its type but delivers no information. Programs can be prototyped entirely with null objects, and when real values are substituted, correct behaviour follows naturally. This is one of many techniques Jules covers in the book for eliminating, rather than merely reducing, bugs.

The Dark Ages of Software

In a chapter on what good code looks like, Jules draws an extended analogy between the post-Roman “Dark Ages” and the current state of software culture. Just as the early medieval period is obscure to us not because it was unpleasant but because its artefacts were not preserved, software lacks a transmitted canon. NASA has had to appeal for working 8-inch floppy disk drives to recover old data; a BBC millennium project produced optical disks that no device could read before the project was even finished. Jules argues that the industry reinvents its values every generation because it has no heroes, no canon, and no culture of reading code for pleasure the way musicians listen to music or architects study buildings.

Writing the Book

Jules credits a decade writing a monthly column for the British programming magazine .exe as the foundation of his writing craft. When it came time to write Extreme Reliability, a weekly train commute between Scotland and London gave him six uninterrupted hours each week to write without distraction. He composed in LaTeX, which he found indispensable for taking a bird’s-eye view and rearranging arguments — eventually cutting the manuscript roughly in half when he realised he had been writing two books simultaneously. He used Claude to generate a first-pass index, which he then refined, describing it as a good example of appropriate AI assistance.

Why Leanpub

After exploring traditional publishing, Jules found that every meaningful conversation with a publisher came only through personal introduction, and that publishers offered uniform style constraints, no promotional support, and royalties amounting to roughly 50 pence per copy on an expected lifetime sale of 3,000 copies. Within five years on Leanpub he had already earned more than the projected lifetime return from a traditional deal. He notes that the primary trade-off is that the author must do all production work independently — “it only takes five months to make a baby” — but considers the creative freedom and financial return well worth it.

Visit Jules’s website at julesmay.co.uk and find the book at leanpub.com/extremereliability.

This interview was recorded on [interview-date].

The full audio for the interview is here: https://s3.amazonaws.com/leanpub_podcasts/video1488727298-edited.mp3. The Leanpub Podcast is available on our YouTube channel at https://www.youtube.com/leanpub, in Apple Podcasts here https://podcasts.apple.com/ca/podcast/Leanpub/id517117137, and almost everywhere else people listen to podcasts.

Transcript

Len: Hi, I’m Len Epp from LeanPub, and in this episode of the LeanPub podcast, I’ll be interviewing Jules May. Based in Montrose, Scotland, Jules is a freelance consultant specializing in safety-critical systems, mathematical software, and compilers and languages. He’s been writing, teaching, and speaking for 25 years, and conducts frequent lectures and workshops. Jules is the author of the book Extreme Reliability: Programming Like Your Life Depends On It. In the book, Jules argues that, and I’m quoting here, there really are techniques and methodologies which will deliver perfectly functioning software at industrial scale. Tucked away in unregarded corners of our industry, there are developers turning out high-quality software in a scalable fashion. Presented in this book is, this is how other great programmers have understood and solved their reliability problems, and this is what we can learn from them.

In this interview, we’re going to talk about Jules’s background and career and professional interests, and what he’s learned over the years. It’s a really great book, and in the end, we’ll be talking a little bit about his approach to writing and maybe teaching, too. So Jules, thank you very much for being on this episode of the LeanPub podcast.

Jules: Thank you very much for inviting me.

Len: I always like to start these interviews by asking people for their origin story. So I was wondering if you could talk a little bit about where you grew up, and how you became interested in the things you’re interested in, and built your career into the career you’ve built.

Jules: Okay, sure. I got my first real proper job working for an aviation company. We were building software systems to control aircraft. Now, this was many years ago. This was very early 80s, and so we didn’t know as much then about how to write software as we do now. But nevertheless, we did know that we needed to be able to build aviation systems that were reliable. We knew even back then that aviation systems have this very interesting property, in that when they crash, you don’t get to switch them off and switch them on again because you have a hole in the ground. And this can be very expensive and do terrible damage to your reputation.

So one of the things that we had attempted was to try to reduce what we were doing down to mathematical terms. For example, one of the things that we had at this time was a chip which had been mathematically proved to be correct. And what was interesting was that we knew that it would work, but we found it was useless for most practical purposes, because you had to disable so much valuable stuff in order to make it provable. So we could make a more complex chip, but we didn’t know whether it worked or not.

So by the time I had come into this, the attention had moved into the software. Could we prove mathematically that the software was correct? And the answer came back very similarly, that yes, we could if we had exactly the right kind of software. And in terms of the kind of software that we were writing back then, no, it turned out we couldn’t prove it. But what we could do is get very close to it. We could create useful systems which we could build with very, very high expectations of reliability.

And the truth is that in the intervening years, since we started putting computers in aeroplanes, not one person has ever been killed by a software fault. Now, the thing about this was that although the project that I was working on wasn’t the complete success that we hoped it was, I did learn an awful lot from it, including a taste for reliability. And I’ve taken that into other fields. I’ve taken this into medical instrumentation. I’ve taken it into automotive. But it also scales up to enterprise scale — the same techniques that we use to keep those aeroplanes flying, we can use in modern software now.

Len: Great. Thank you very much for sharing that. That’s really great. I have a couple of questions that I’m going to ask you specifically about that later on. We have already done a launch video for this book, and we talked about the truth that no plane has ever crashed because of a software fault. And I’m actually going to ask you to talk about that a little bit again, just because it’s so interesting and sort of contrary to what I think a lot of people think. But before we do that, just one question I have for you is — so over the years, I’ve interviewed a lot of people of different vintages, and partly because so many people that write books published on LeanPub are in computer science and working in programming and software engineering and things like that. You studied computer science in the late 70s and early 80s. What was it like? Things have changed so much. I imagine every five years or so, the experience of studying computer science would have changed enormously in the last 50 or 60 years that it’s been taught. What was it like for you?

Jules: Yeah. I’m very grateful for my education. In fact, I selected university — I selected Warwick in the UK — because they actually had a reputation for theoretical developments. This was one of the places where functional programming was being developed. We had a language called Waffle, which was very similar to a language called Sassl, which eventually turned into Miranda and eventually turned into Haskell. So the foundation was being built there.

There was a department which was concerned with graphics systems, which were real exotica back then. I mean, it took an extraordinary amount of memory just to build a frame buffer. But we had people there experimenting with things like not just generating images, but also computer vision. There were people there working with speech understanding and speech synthesis. They weren’t calling it AI back then. They were taking much more deterministic approaches, but people were doing it. And we had some very interesting results from that.

What it was, was highly theoretical. I joke that it’s just another kind of branch of maths that they were teaching back then, but I am grateful for it because it was very mathematical. Obviously, there were parts of it which were about programming, but it extended everywhere. So in the very first term of the very first year, we had to build ourselves a processor out of discrete logic. And then we started to program onto it. And then later on, we wrote compilers for it. And so they gave us sight of the entire ecosystem. And they were expressing this in these very theoretical, mathematical, engineering type terms.

Today, it’s much more about programming. Today, I think what universities want to do is to put people out who are employable directly in a programming shop. But of course, back in the 70s, there wasn’t such a thing as a programming shop — we didn’t have the scale of it at that point. But what I found is that the stuff that they taught us back then — the basics of language, the basics of computer languages, the basics of computation, the basics of complexity, even the basics of arithmetic, which I thought I understood but turned out I didn’t — this has set me in really good stead for a half a century career. And these things just don’t go out of fashion because the basic laws of mathematics, the basic rules of engineering haven’t changed very much.

Len: One question that’s come up on this podcast many times over the years, partly because of the various vintages of guests, but partly because so many people that write books published on LeanPub are in computer science and working in programming and software engineering and things like that — one of the questions is, if you were starting out now with the intention of having a career in software, would you do a four-year computer science degree? And it’s interesting because what you’ve described about how those degrees may have changed over the years may actually make your answer a little bit different from most people’s.

Jules: Okay. I’m going to give you an answer which probably nobody else has ever told you. I would say yes, I would do it. I would go back and I would do the four-year degree probably 10 years into my career. And the reason that I would say that is that software is actually an enormous field. The way we see it most of the time, the loudest part of our profession are the people who are building websites and e-commerce systems. And when I say to people I’m in software, the first thing they do is they say, oh, do you build websites? Because that’s how people — including the people in the industry — see it.

But there’s a great deal more to it. When I started my degree, I didn’t understand how much scope was available to me. But having got started and having got a bit of perspective, I was starting to see it. And if there’s one thing I’ve learned, it’s that there is a space in this industry for everybody.

I was talking to my doctor a while ago and he said the thing that he loves about medicine is that it’s so huge and there’s a space in it for everybody. And he was describing this person that he knew when he was going through his degree. He said this person had the worst bedside manner in the world. They were intelligent, they were skilled, but their bedside manner was such that they could make even healthy people sick. They went on to become a brilliant pathologist.

I feel that way about software — that it is such a huge field. We’ve got people building these large systems. We’ve got people building these tiny embedded systems, the things that fit on one chip. We’ve got people building the infrastructure and the networks that make everything else work. We’ve got people who are working right down at the hardware, the things that make the screens that you’re looking at work. This is all software, this is all stuff. And I think that if I was starting now, I’d take 10 years to give myself enough perspective to understand what it is that was really drawing my interest.

Len: Thank you very much for that really great answer. I’m very sympathetic to that kind of answer myself. I’ve spent 10 years in university — for my sins, it was English literature. And one of the interesting experiences I had was being a teaching assistant when I was doing my master’s for an engineering class. These engineers were forced to take English literature in their first year, for very good reasons I think. But one of the interesting experiences I had there very directly was people saying, how am I going to use this? This is useless. And I remember being quietly infuriated, thinking — I was only a few years older than them at the time — how the hell do you know what you’re going to need to know in your job? If you think your degree is training for your job and you haven’t finished it yet, why are you so confident that you know now what you’re going to need to know at the end?

And in a way, the emergence of what we now call AI in the last couple of years in programming has been a great comeuppance for people who had that attitude. What I see in books that people are writing now is that what matters is how deep the knowledge is that you’ve developed. That’s what allows you to compete now. Everything you learned with respect to your job as a programmer ten years ago doesn’t really apply a lot now in terms of the day-to-day sitting in front of a desk. And really what people are throwing back on is, what do I actually understand? What do I actually know? How wide is my knowledge? And how great is my imagination?

Jules: I agree totally. I think it’s a mistake to think that universities, that the academic system, is supposed to prepare people for jobs. That’s what apprenticeships are for. That’s what on-the-job training is for. Academia is for something else. It’s for developing that deep knowledge. It’s for pushing the boundaries of knowledge. It’s for being explorative. And these are not things which industry values.

Now, industry constantly complains that universities are putting out people who are not equipped to do a job. Well, they’re not supposed to.

Len: I mean, yeah, again, I couldn’t agree more. If companies want free job training from the government, they’re welcome to desire that. But the idea that you should be paying for your own job training and that the company should complain that after you pay for your own education, you show up and you’re not ready — that’s ludicrous. And especially because we all know that things change in any industry so quickly that if you let CEOs of companies tell universities what to teach, students would start their curriculum, they’d end four years later, things would have changed so much, and the CEO would be complaining that you don’t know how to do your job — and they wouldn’t even remember what they said four years ago.

There have been initiatives like this in the UK in the past. There have been initiatives like this in Canada — for example, in the province of Ontario, the province is considering having rewards for universities and punishments based on how students are employed after they finish their degrees, specifically based on whether you got employed in the area that you studied. And it’s like, that’s not — these are presented as kind of real-world, no-nonsense measures, but they’re the most insane, hallucinatory garbage that you could possibly have.

I mean, who are some of the most famous technologists in the world? Bill Gates, who dropped out of school — was that the school’s failure somehow? Did Elon Musk study automotive engineering? No. If you want to pick the people who are presented as the serious paragons of no-nonsense business, nobody studied what they ended up doing as a subject in university. Anyway, there’s my little rant. What it’s about is learning things and really understanding things, and you have to do that yourself.

Jules: I don’t know how it is in Canada, but certainly over here, between 60 and 70 percent of graduates go into a completely different field from what they studied. They’ve been successful in their studies, but they’ve gone into a completely different field. Now, I don’t see that as a failure anywhere. You could say that it’s a failure of intake — maybe they recruited the wrong people into their courses and didn’t screen them properly. You could possibly frame it as a failure of the education system in that people who went through high school didn’t understand what the tertiary education or what the industrial version of what they were studying was. But I don’t see that as a failure at all. I think that, you know, you take somebody who’s got a law degree and you place them into an advertising agency, or you take somebody who’s got an arts degree and you put them in a technology company designing interfaces — these are the right things to do. It allows you to get this sort of cross-fertilization.

If I have a criticism, it’s that British degrees are too narrow compared to, say, American degrees, which are more generalist.

Len: Yeah, I mean, I did my doctorate at Oxford and it was there that I learned about the single-subject degree. I didn’t know about that before. In Canada and in the United States, there’s a kind of liberal arts tradition that informs the undergraduate degrees where you’re forced to take a number of electives — the forcing part being the most important part — because at 18, how do you know what you need to know? People don’t like to hear that, but that’s the way it is. You don’t know what you don’t know until you know it, which is just the nature of what it means to become educated.

The argument obviously is that if all you’ve studied is English, you’ve got a deeper understanding of English than if you’ve been forced to take a wider range of courses. And yeah, I couldn’t disagree more with that. The novels that you’re reading — the authors who wrote them weren’t restricted by anything at all, and any knowledge you have of the world can inform that. So thank you very much for letting me go on about that, and thank you for talking about your own very informed opinion. That’s great.

So the next question I have is just talking a little bit about your career. Specifically, working in aerospace — I think a lot of people, when they think about a programmer, think about e-commerce, they think about websites. What was the day-to-day work like? Did you have a desk with a personal computer connected to a network? Were there a lot of whiteboards? How did it go? This is the early 80s, right?

Jules: No, we didn’t have personal computers on the desk. We had a computer — a very large computer — in a hermetically sealed room inside a Faraday cage. I mean, they were so worried about security. And you will understand from that that I’m limited in how much I can talk about this. But we had a tea lady pushing a trolley up and down the corridors. It was at that level. It was kind of medieval.

There was a lot of writing. One of the most used pieces of equipment was the photocopier. And one of the things that would happen is that every week we’d get a dossier that would come around. Each one would be addressed to each person and only we were allowed to open it. These would be the errata for the files that we already had on our desk. So somebody had a central record and data would change for various reasons. And so what we’d have to do is spend an afternoon merging these corrections into the files that we already had. These were things like technical specifications, conversations that people had had, policies — all sorts of things. But everything was recorded in physical paper files.

Len: That’s really amazing. So were you writing code?

Jules: The main part of the job was not writing code.

Len: Oh, okay.

Jules: And part of the job that I had at that point was applying these techniques. We were using a language called Z to express — you’ve seen that AIs are now writing mathematical proofs in a language called Lean. That is a sort of pre-version of Lean. It allowed us to talk about computation and truths about computation in a formal way. And so what we were doing was using this language to try to understand the flight code that we already had. And a lot of our output was not necessarily the proofs, although that was important. It was — well, are these proofs useful? And how do we assess what’s going on here? And can we transmit our experience to other parts of the organization? So a great deal of it was this kind of commentary, what you’d call analysis.

But we did nevertheless keep our hands in programming. We had new equipment coming along and we wanted to learn how to use it. So we had a graphical computer — again, very early, and remember at this point frame buffers were incredibly expensive. They used a large proportion of the world’s memory, for example. But we wanted to understand how we could use these, so we were writing test code onto those.

Len: Well, yeah, thank you very much for that great explanation. That’s not the image that most people have of programming, or of being on top of a kind of software system and doing that kind of analysis. And it makes sense — if you know that the things you’re building are things where people will die if they don’t work, it makes a difference that you’re doing things that way.

So I guess the next thing I’d like to talk about, just before we move on to talking about your book, is you mentioned that no airplane has fallen out of the sky directly because of a software fault. We talked about that in the launch video. And I was wondering if you could talk about that again, because I personally had been under the impression that that had happened not too long ago. That’s not true, and so I would like to ask you to set the record straight for those of us who read the headlines wrongly and got the wrong impression — and what it is that actually has caused planes to fall out of the sky where people believe that it was software problems.

Jules: Okay. Most of the time, let’s talk about the engineering, the mechanics of an airplane. An airplane is normally a very stable system. The way they design aircraft — and particularly, this was a 737 MAX, and that was a classical passenger-type, civilian aircraft — so if you just take your hands off the controls, then all the control surfaces level it out and it will just go into straight and level gentle glide, even unpowered. If all of the engines fail, you can glide that plane down on stick power alone, because it doesn’t require any electronics to actually function.

Now, it’s also probably worth saying that in those big aircraft, there are two schools of thought. There is the Boeing school of thought, where you want to put as much control in the pilot’s hands as you possibly can, because he is in charge of the plane. So Boeing cockpits are very complex, and the pilots do require an awful lot of training. On the other hand, there’s the Airbus approach, which has been characterized as a cockpit containing one switch — one position is up and the other position is down, and that’s all you need, and the plane flies itself. Now, Airbus thinks its approach is better because it requires the pilot to do less work. Boeing thinks that its approach is better for the exact same reason — it requires the pilot to do the work.

So the 737 MAXs — they were Boeing’s planes. Now, there is another thing that’s important to understand. As the plane’s flying along, if you nose up, you’re trying to get lift off the wing, which lifts the aircraft up, but as you’re doing it, you create more drag and more turbulence behind, and there comes a point at which the air detaches from the wing and the plane starts fluttering and you lose the lift. That’s called stall. And every aircraft, from the smallest ones to the largest ones — as a pilot, you’re trained to feel the onset of the stall, and then nose down again.

Now, in the 737, they simulated that. They said that as the plane was nosing up, before the stall happened — because that would be a very serious thing in a plane that size, particularly when it’s fully loaded — before the stall happened, it introduced this flutter so the pilot could feel it, and then, if the pilot didn’t respond, it put the nose back down again to bring it back into straight and level flight. This was detected by two sensors just underneath the fuselage called the stall detectors. Basically, they’re looking for the direction that the air flows. So what happens is, when the stall detectors detect that there’s a stall likely to happen, it flutters the stick, and then it pushes the stick forward to bring the nose down.

Now, what happened in those two crashes was one of the stall detectors was broken and it wasn’t producing a signal, and the other stall detector was doing all the work. And so what happened was, if anything happened to that, then the plane thought that it was now in a stall situation, so it would nose down. And what happened in those two crashes was the second stall detector was reading wrong data for various reasons, and the plane pushed the nose down — but it now pushed it into a dive, which is also a very serious thing.

Now Boeing, being Boeing, they’ve got a button on the yoke that says yes, I know the stall arrest has happened — you press the button and it relaxes and the pilot has control. But because the detector was still failing, the plane was still trying to nose down.

So here it is — all of the software was doing exactly what it was designed to do. It was designed to keep the plane safe even when the pilot was being a twit. It was also designed so that if the pilot really did want to take control of the plane, he could do it. It’s a button on the yoke, it’s right there under his thumb.

But what happened in those two cases — and they were both the same thing — first of all, there was a maintenance log which showed that those stall detectors were malfunctioning, and had been for in one case a month, and nobody had done anything about it. So the aircraft — and it shouldn’t have flown that way — had a known defect, followed by another defect in flight. But the second thing is that in both cases the pilots weren’t trained. In one of those cases, the pilot had never been type-approved on that aircraft. He was effectively flying without a license.

Now, that wouldn’t normally have been a problem, but — Boeing would normally have provided a book in the cockpit: if something’s going wrong, you open the book and the book tells you what to do about it. And the book is correct, and had they opened the book, it would have told them what to do about it. But Boeing made a mistake in that they installed this stall-arresting software in the plane but didn’t tell the pilots that they’d done it. So the pilots didn’t know that this software was operating.

So you had a whole series of mistakes building up. You had a lack of maintenance, a lack of training, a lack of proper type approval coming from the manufacturer — so there was a whole bunch of things that went wrong, which meant that when the software did the one job it was designed to do correctly, it was pushing in the wrong direction. Is that making sense?

Len: Yes, no, that’s totally making sense. Thank you for that very great explanation. One thing I believe — just to add to the complexity of the situation — I believe you mentioned in our launch video that in one of those crashes, the co-pilot understood what the right thing to do was and told the pilot about it, and the pilot said something like, you shouldn’t speak to your elders that way.

Jules: Something like that — I can’t remember the exact words. But yeah, it’s a problem particularly — and I know this is going to sound racist, but it’s particularly in Asian countries — Japan, Philippines, China, places like that — there’s this strong culture of respecting your elders, so it’s very difficult for a junior. The co-pilot is supposed to be there to keep the captain honest, but if he’s subordinate, it’s very hard for him to do that culturally. Europeans don’t have that problem in anything like the same way. Americans do, but not as badly as the Asians do.

Len: Sure, yeah. I wouldn’t say that’s racist at all — all cultures have their own features. I mean, there are cultures, for example, where the opposite might be the case in a sense, where recklessness is rewarded rather than authority. And you can just imagine all kinds of cultural failings happening just because of their own nature.

I was once on a flight into Houston and as we started the descent, the pilot came over the intercom in the aircraft and said we’re going to land in about five minutes — yeah, no, I mean, it’s a generalizable thing that cultures actually exist and they have their own characteristics, and they all have their own weaknesses and failings.

It’s interesting that, you know, when you describe the kind of Boeing versus Airbus difference there — there is a kind of experience that a lot of people are having actually, to go back to software engineering nowadays, that’s kind of like that too, right? You know, YOLO — just tell Claude what to build, Claude builds it, merge it, deploy it, see what happens. And you just kind of trust it to do the right thing. And the other approach is — wait a minute, no. You still use the technology that’s available to you, sure, but someone actually needs to be there, there still need to be fail-safes, and someone needs to know what they’re doing underneath it. Otherwise you’re going to encounter problems, and you might encounter them too late to fix them, at least for some people.

So, moving on to the subject of your book, Extreme Reliability. Just to kind of set the stage — one interesting thing, when we’re talking about software, we think we’re talking about people or about machines, but we’re actually talking about people. People design them, including AI systems — people design those too.

One example I often bring up when I complain about software — which everybody does nowadays because we all encounter it all the time — is there’s a certain payment service that we use, and years ago I logged into it one day and all of a sudden there were these circle faces in front of me and a little prompt saying, who would you like to pay today? And it’s like, that’s not what I’m doing here — I’m not joyfully opening my payment system to cheerfully brighten someone’s day. And all of a sudden a lot of the problems that I’d had with that payment service were explained to me negatively by noticing that nobody who had ever, in their personal life, in their vicarious life, or in their imagination, been confronted with the fact that if you sent $50 to the wrong person instead of your grandma, there wasn’t another $50, and grandma couldn’t get her bus ticket to come visit for Christmas — nobody behind any of the decisions that led up to those cheerful circle faces on the payment system had ever been anything other than well off in their life.

So I guess my first question would be — when it comes to reliability in software, is there a selection process that has to happen for personality type?

Jules: That’s a really interesting question. Because that’s kind of a different question to the example that you laid up with it. And I think I’d like to answer them both.

You are right that people who are inside an organization have a tunnel vision. It doesn’t matter where you are. They have their values and they have their way of seeing the world, which are absolutely unique to them and are completely different from that of the public. And I’m sure you have seen this. I’ve been inside lots of organizations which go through a process of eliciting their values. And they always come out with values like integrity and innovation, as if this is what makes them, this is what distinguishes them from other people — as if nobody ever says, well, integrity is not one of our values, we are going to exploit our employees and ream our customers. Nobody says that, right?

There are values which are so obvious to anybody from the outside, but which they can’t articulate because no fish ever defined water. A good example of this was when I was at Adobe and they went through this value elicitation process and got all of the usual anodyne crap. But the thing which characterized that company was that they were design-led. Everything about that business was designed. It was visual. It was a visual treat. They ordered an office while I was there, and the first people that moved in were the interior designers. These people had a health and safety video that you actually wanted to watch twice because it was so good. They didn’t think to define this as one of their values, even though it was perfectly obvious to everybody else.

So I think that’s really important. I think that a business, and by extension the developers inside the business, need to be sufficiently self-aware to understand what it is they value and what makes them unique. And that goes a long way towards understanding the extent to which you can be miscommunicating with your public. It’s not a bad thing to have these values. It’s not a bad thing to have these cultures, but you do need to be able to calibrate them.

The second thing is that you also have to be able to understand your customers in those same terms. One of the guys reviewing my book sent me a comment at one time. He picked up a thing that I’d said — that the thing that we’re trying to do with the code is to produce a solution, to actualize some kind of technical specification. He said, no, no, absolutely not. What we’re trying to do is to keep the customer happy, deliver value to the business. But the trouble is that those two things are often a different thing. Because what happens if a business says, we’ve made the decision, we’re going to produce this piece of functionality because we think this is what our customers are going to value, what they’re going to want, like those smiley faces, and the development team says, so what planet are you on, where did that come from, we’re going to go broke doing this — who’s going to win that argument? It’s not the programmers.

So the programmer’s customer is the business, and the business’s customer actually is the shareholders. And the shareholders, if they’ve got any sense, their customer is the customer — and probably it’s not. So in order to create a good product, you do have to be able to square that circle. You do need to understand, as a business, how what you’re delivering to your customers matches with your own values. And that requires quite a lot of introspection, I think, which a lot of people don’t do.

Len: Definitely, definitely. Yeah, thank you very much for that. Really great. I’ve never quite thought of it that way before myself — that it’s the shareholders for whom the customers are, in a sense, the kind of customers. That’s very interesting.

When it comes to software reliability — so let’s move on to talk directly about your book now, Extreme Reliability — one thing you talk about is how basically code doesn’t have to be perfect to work perfectly, which is an intriguing way of expressing the idea. Can you talk a little bit about what that means?

Jules: Yeah, sure. You will imagine I have spent the best part of 50 years talking about how to make reliable software. I have a background of making software that never goes wrong, and then I take that out to the real world and the real world tells me, but you can never make it perfect. I will always find a way to make it break. You’re absolutely right, you will. Programs are fallible. And even if we weren’t, even if we could build perfectly working software, we can’t control the environment, and the environment is going to knock it off course.

But that does not matter. We started off talking about the airplanes, and I said that those big airframes — even the small ones — they’re designed to be inherently stable. The engines can fail, the pilot can have a heart attack, and the plane will go into a straight line naturally because it’s designed to be stable. It’s very hard with modern airplanes and the kind of safety interlocks they’ve got to put it into a situation where the plane will damage itself. So it doesn’t have to be perfect. You can have failures, but the plane will still behave more or less as it’s supposed to behave.

Now, it’s worth saying that the number one task of an airplane is to not kill its passengers. The number two task of an airplane is to get them where they want to be. And it’s important to keep those in that order. And again, with things like software systems, we don’t always do that.

But if we built software along the same lines — if we said, okay, we accept that the software is imperfect, we accept that the environment is imperfect, now what do we need to do to make it keep going in spite of that? — then that gives you, if we design our systems to be intrinsically stable, it doesn’t matter if they’re imperfect. What matters is that they’re not so imperfect that it pushes them outside their range of stability, like a boat that won’t capsize. You push it so far and eventually it’ll bob up again.

Len: Yeah, no, that makes a lot of sense. It’s interesting. I think one thing that is hard to wrap one’s head around is that software actually can be reliable and work over time. And I say that as someone who is the non-programmer at a company of programmers, but also as just an ordinary person who reads the news and uses things. We’ve got an example in Canada now, a federal government civil service payment system that they’ve spent like a couple of billion dollars on at this point, that notoriously doesn’t work. So when you’re at the pub or at dinner with your in-laws or something like that, and I’m sure you get the question — so you’re in software engineering, why do things break all the time? — what’s your answer to people who ask how these big projects that have billions of dollars spent on them can fail and be unreliable?

Jules: Okay. There are lots of answers to that. An example in the UK, of course, was Horizon, which was this Post Office system.

Len: Yes.

Jules: Which people went to prison for, people killed themselves over. It was a terrible, terrible situation. But one of the things which made that so poor was, first of all, the programmers were incompetent. I remember seeing a piece of code to take the absolute value of a number and it ran to about 10 lines. They didn’t even realize there was an intrinsic. It was just really crappy code.

But the second thing was that it wasn’t built according to the domain principles. We have known since the 13th century how to build robust accounting systems. Everybody in the world does it. Horizon didn’t, which meant that when the operators were reaching in and tampering with it — sort of putting God hands into it, which was necessary because the system itself was unreliable — this interference left absolutely no trace. They were effectively rewriting history and nobody could tell the difference. So that’s the first thing— incompetent programmers. And the second part of that was incompetent domain analysis. You have to build on the knowledge that we already know about. You have to use the language of the domain that you’re working in.

But the third thing, which I think is the more important point, is that I was just spending some time talking about stabilities. There are some systems that we have which are unstable. If you take a wine glass and you hit it, it’s going to shatter. If you kick a football through a window, it’s going to shatter. And it shatters because of the way glass works. When you put force on glass, it eventually cracks, but then it transmits the energy to the end of that crack. So the crack starts moving and then it splits and it moves. And so once the glass starts cracking, generally speaking, it spreads and it shatters. And it does that catastrophically. Unlike an aircraft, unlike the boat.

But that’s like software. We actively design our software to be brittle like glass. Imagine if we built our pushbikes or our cars out of glass. Imagine how dangerous they would be. But this is exactly what we do with software. And this is how we do it, and we’ve been trained to do this for as long as I have been in this business. We’ve been trained to proceed, do your computation until you reach a point at which something’s going wrong. For example, you want a piece of data. So you go and open the file and the file fails to open. Now, at that point, we treat that as an error. We say something has gone wrong because the file has failed to open. But it’s not an error. It’s perfectly reasonable that the file has failed to open. The file might not be there. Somebody else might have it open. There’s any number of reasons why the file might not open. This is not an error. This is not an exceptional case. It is an exigency. It is something that is true that you’ve got to deal with.

But what we do is at that point, we say, all right, we’re going to declare an error. In the old days, we would hand back an error code and we’d force the code that called that to say, okay, I’m now going to have to deal with this error. I’m going to have to stop doing what I was doing and start doing something different. Today, we do something even more destructive — we throw an exception, which means somebody over there somewhere that we don’t know about is going to have to deal with the error. But the point is, we stop doing what we were trying to do and we now throw the program onto a completely different path. And what that path does, in both cases, you can’t fix the problem. You can’t carry on as normal because you’ve thrown away your context. You’ve thrown away the reasons why you got into that in the first place.

So what you’ve done is you’ve got something which is going off the rails over here, but we’ve amplified it. And now it’s going even further off the rails. And then we do it again and we amplify it again. This is brittle. This is brittle software. This is why when we make our software, the tiniest thing going wrong in the environment or the tiniest thing going wrong in the program makes it go off the rails, makes it crash. Look at null pointers, for example — one single null pointer, one slightly malformed value somewhere in some irrelevant backwater in the program can bring the entire system crashing to a halt. This is unforgivable. More than that, it’s stupid.

So that’s why software is unreliable — because we build it to be unreliable. We build it to be brittle. You can’t just take your hands off it and let it get on with itself.

Len: One thing you brought up earlier is who the customers are, as we spoke about. So the programmers’ customers are basically the executives who are sitting on top of the business in a sense. And so I imagine if some executive were walking through the software engineering department and they saw a book called Extreme Reliability, they’d think that’s extremely expensive. I’m sure you’ve encountered this before. What would you say to someone who’s read your book? What can they say to that stakeholder when they’re in the meeting saying, we need to introduce extreme reliability into our software, and the concern comes back — you know, perfect is the enemy of the good, etc., etc. — all those kinds of things that people would say. What would you arm someone with in that conversation?

Jules: Okay, well, the first thing is, perfect is the enemy of the good, but I don’t care because it doesn’t have to be perfect. It just has to be good enough. But what’s the most expensive thing? When you’re speaking to the manager, when you’re speaking to the bosses, then they are speaking the language of finance. They’re saying, what’s expensive? Well, I’ll tell you what’s expensive. It’s having a major failure. It’s not being able to deliver your core value. It’s sending money off to somebody who’s not your grandma. It’s leaking all of your customer data. These are bloody expensive things, right? You better apply some attention to that. Now, normally they don’t. Normally, they will leave that to the skill of the programming team.

But the other thing that I’d say is that when the bosses come down and they say, okay, we want this — one of the properties that software has is that the size of the change in domain terms is often completely disproportionate to the size of the change in software terms. Sometimes somebody will say, well, this is a big change — and you say, okay, it’s done. But more likely, they say, it’ll only take you five minutes, and it takes you six months. And this is completely unpredictable.

But if you could say to your bosses, no, it’s all simple. Whatever it is you ask for, I can pretty much guarantee it’s going to be easy to do. And if it’s not, it’s not going to be disproportionate. Because that’s what safe software gives you. You are not going to get shut down by bugs. You are not going to get shut down by security faults. Because remember, every security violation is a failure of reliability, of course. These things are not things that are going to shut you down.

One of the things that I do is I work out a mathematical model of technical debt in terms of what it costs, why technical debt is debt, and why it compounds. And one of the things that I’ve worked out is that even in an absurdly high-functioning team, you have typically four programmers doing the work of one. And all of the rest of it is servicing the technical debt. Now, if you take that out, then — you can’t obviously get rid of all of the technical debt, but you can multiply the velocity of your developers by three times. That’s a good thing. And in fact, in reliability terms, the reason why these reliability techniques deliver that is that you’re not fighting the technical debt all the time. It’s quicker and easier to do a job right than to constantly rework it.

Len: And on that note, just to get specific, what is an example of a reliability technique that you talk about in the book?

Jules: Well, there are all kinds of things, but one of them — I mean, I talked about the things which cause the reliability issues, and one of them would be null pointers. But we almost mechanically solve that by replacing them with null objects. Now, here’s why this works. When you have a type, when you’ve got a variable of some type, that variable has a class that defines an interface. It says this is what you can do with this. So no matter what object you put into that variable, you can speak to it in a certain way. But if you put a null into it, then you can’t speak to it that way. The null pointer has a completely different interface. It doesn’t do any of the things that you thought the type did. It might do a few other things instead. But fundamentally, every time you use that variable, if it might be null, then you’ve got to say, well, is it really one of these or is it really one of these? People forget to do it. And of course, if it turns out that it is a null, then it’s an error and you can’t carry on on the path that you were on.

But if, instead of that null pointer, you replaced it with a null object — now the null object is a thing which exposes the entire interface. It does everything that it’s supposed to do, but it delivers no information when it does it. So think of a string, for example. A string — you can ask how long it is, you can capitalise it, you can append things to it, things like that. A null string has no length. A null string has no content in it. If you try to print it, then there’s nothing there to print. If you try to capitalise it, there’s nothing to capitalise. So this is something which behaves in every respect like a string, except it’s dead.

Now it turns out that you can then propagate these around your program ignoring whether it’s null or not — the program just works. You can use these things as fail-safes. You can say, if you can run your whole program using nothing but these null objects, then when you start putting real values into them, the program is going to work exactly the way you thought it was supposed to. So this is a stabilizing process. It’s also a great prototyping process. There are all sorts of things that you can do with these null objects simply by replacing the null pointers.

Len: Thank you very much for that very specific and very clear example of a reliability technique. That really makes a lot of sense.

The last thing I want to ask you about when it comes to your book specifically — you’ve got this section I’ve got open in front of me here, this chapter called What Does Good Code Look Like? Can you begin by talking about the fall of the Roman Empire and the emergence of what used to be called the Dark Ages? I just want to give people a little bit of a flavor of your approach to explaining things. So why is it that you talk about the Dark Ages and the Renaissance at the beginning of a chapter about what good code looks like?

Jules: Okay. That’s a great question. Western history — we all know about the Romans. We all know about how the Romans spread out. They covered most of Europe. They went way, way out east. And then the empire collapsed. And yet we still teach Latin in universities. My other half is a lawyer. She had to learn Latin to read statutes that are still on the books today. Somehow, the Romans have transmitted an awful lot of culture. The language that we speak owes a lot to classical Latin. The values that we hold, a lot of it dates back to the Romans. It was a very influential time.

But the Roman Empire collapsed, disappeared between about 800 and 600 — something like that — and it just shrunk back to nothing. And then there’s this sudden surge of knowledge back in the 1600s. This was the Renaissance. This was when people started writing. It’s when what we think of as science really started. And again, a lot of things that are influencing us today came from that time.

So what happened in the interim? Was nothing happening? Were people sitting around for like a thousand years saying, oh, I wish something would happen? These people were living generally peaceful, healthy, productive lives. There was farming, there was trade, there was writing, there was culture. There was an awful lot of stuff going on. It was not a bad time to be alive. But we don’t know anything about it because they didn’t write anything down. They certainly didn’t write things down in stone the way the Romans did. They built buildings out of wood and paper and mud. They didn’t build them out of stone. We know very little about their music. We’ve got almost no musical instruments from that age. Some, but not a lot.

Where I live in North East Scotland, it’s famous for the Pictish environments. We know what their written language looks like. We have no idea what it sounded like. We can’t translate any of the stuff that we can find. This was in the Dark Ages. And it’s dark not because it was a sad place to live and the sun didn’t come up. It’s dark because it’s obscured from us. We can’t see it. The people who lived in it were living good lives. But when we look back at it, there’s nothing there to see. It’s just a shadow. It certainly wasn’t the Game of Thrones that people think of it as. It was okay.

Well, moving forwards now — I know that NASA, a few years ago, put out an appeal for anybody with working 8-inch floppy disk drives, because they have a load of software sitting on 8-inch floppy drives and they don’t have any hardware that can read it. Before then, I remember the BBC put a project together. It was in celebration of the 1,000th anniversary of the Domesday Book. They were going to make a new Domesday Book. And so they were going to create an optical disk, like a 12-inch CD, which contained an awful lot of what we knew about the world. I mean, not everything, but it was trying to be completely encyclopedic. And it was a massive, expensive project. And by the time it was finished, there was not one device in the world that could read one of these disks. The whole thing was lost before it was even completed.

Now, when we look back — our jobs right now are to produce code, and we store it on ephemeral media. I’ve got disks in this office which I can’t read because I have no computers that can use them. Even if I had the disk drives, I can’t put them into the computers. That software targets hardware which is now obsolete. And everything that we do, because of the way this technology moves, we target this ephemeral software onto obsolescent hardware all the time. We have no history.

We are living in the exact same dark ages as those people were a thousand years ago, because we are not transmitting our artifacts to the next generation. We’re not even transmitting our values to the next generation. The number of times I’ve seen shitty 20-year-old ideas regenerate, and we all look at each other and say, you know how this story ends, right? But we literally don’t have a culture or a history.

And the proof of this is — I ask people, so who is your favorite programmer, or favorite developer, or favorite computer scientist? The person whose code is so good, you read it just for pleasure. Not to steal a snippet or to debug it, but to say to your friends, hey guys, look at this, this is so good. Who? And of course that’s a rhetorical question. But can you imagine a writer who says, well, I write books, I don’t read them — why would I want to read books? An architect who never looked up. A painter who never goes to galleries. A musician who says, I play music all day, why would I want to listen to more when I get home? These are absurd.

And yet we consider ourselves to be a creative, fast-moving industry, and we don’t have our heroes. We don’t have our canon. We don’t have anything that corresponds to a Bach or a Mozart or a Van Gogh. We don’t have any of that. So every generation, we have to rediscover, reinvent our values and our culture all over again, just as they did in the dark ages.

Len: That’s a very great answer. I’m going to spend some time thinking about that a lot. It never occurred to me before to think about that in the software world. I mean, that idea of having a Bach or a Mozart or a specific person like that. Of course, there are some programming books that are kind of classics that people do talk about — if I could pick the classic book, it would be this or that or the other thing — but I’ve never heard anyone talk about some code that way at all.

So thank you very much for that very thought-provoking answer. And it’s just a great example of the depth that you engage with in your book about starting with simple questions like what does good code look like — that’s an interesting answer there, which is, can you think of an example yourself? It seems like a straightforward question, but can you even answer it yourself?

And just moving on to the last part of the interview, briefly, where we talk a little bit about your work writing. So there are a lot of people who would like to write a book, they’ve got a lot of ideas, but their question is, how do I do it? And so my question to you is, how did you do it?

Jules: Okay, I was lucky. Many years ago, when there was still such a thing as print magazines, I got a job writing a column — a monthly column — in a British programming magazine called .exe. And I stuck with that for 10 years. And I had, at the time, a very, very supportive editor. He basically helped me to learn to write and he helped me to develop a style. And after that, when people met me, they’d say, oh, you sound exactly how you write. So that was really gratifying. I’m very, very grateful for that experience. And in fact, the editor then, Will, he copy edited this book for me to make sure that it was right. So we’re still friends 40-odd years on.

Now I was writing a thousand words, plus or minus about 20, every month. And so I got very good at writing to that length. And that sort of translates into blog writing. And I kept on starting blogs and stopping them — not having a deadline, I found that I wasn’t very consistent at it. And that was one of the things that the magazine gave me. It was a print magazine. The deadlines really meant what they said they were.

So then I started writing more books and they tended to be very small books. There are still ones kicking around on Amazon, which is about how to innovate and how to understand an idea. It’s only very, very small compared to this one. And some books I started and abandoned. But I think the point that I’m trying to make here is I just practiced. These almost writing caters — I’ve got used to this thousand-word format. What does 20,000 words look like? What does 50,000 words look like? What’s the process? I knew what a thousand-word argument looked like. It had a sort of a rhythm to it. I got used to that. But when you start expanding that, then what’s the rhythm like differently? And I started looking at other things. I started going from listening to pop songs and then listening to symphonies. I spent time making adverts — that was a 30-second narrative — then building up to five minutes, and then understanding a movie script, just deconstructing and understanding. Someone got the sense of rhythm.

One of the reasons that I came to LeanPub was that I wanted my voice to come out in the book. And I wanted the rhythm and the narrative to have a particular shape to it. And so I came to LeanPub because I couldn’t find a commercial editor that would let me do that.

When it was time to write this book, I had a really good opportunity. I was living in Scotland, but I was working in London. And so I was taking the train up — commuting every week. And so I’d take the sleeper train down, and then I’d take the day train back up again on a Friday. And I had six hours to myself, not being interrupted. And there’s no phones because the mobile network in those days was awful. So no phones, nobody pestering me. I could just sit down and I could write.

And I would set myself the task of writing for four hours every week. Not really trying to get the ideas in any kind of order, but just getting the stuff down on the page and seeing what I’d got. And that wasn’t my normal style. I was normally used to plotting very carefully, but I found I was just sort of flinging spaghetti at the wall and seeing what stuck. And gradually out of this, two things happened. First of all, the argument started to describe itself and it came to focus. And so I started moving things around a bit. And the other thing was that I realized this was getting far too long. And actually I was trying to write two books at the same time. And so I actually cut half of it away and cleaned it up again.

I started off writing it in Word, but decided that Word was not the tool that I wanted because it didn’t let me think about things and it didn’t let me push ideas around. I know friends had written books on Google Docs, but that was appallingly slow and couldn’t work. So in the end, I switched to LaTeX. It took a while to get my head around that, but I found that it was a really good — it’s the best tool I’ve got for thinking with. Everybody says LaTeX is what you finish on, but no, I couldn’t have done that. I needed that kind of tool to learn how to take a bird’s eye view and push things around and then make it fit together again.

Len: Yeah, thanks for that great answer. I mean, you answered sort of like 10 questions that I would have asked in that answer, particularly — I love the idea of turning your commute into writing time, which is something that I’ve heard from different people. Some people turn their mornings into writing time. Some people turn their evenings when the family goes to bed into writing time. But turning commuting into writing time is a great thing to do as well. And that experience of moving mentally from structured, ordered, planned-in-advance, carrying-out-a-plan to just kind of getting your thoughts out there is actually a story I’ve heard a few times as well. It’s sort of funny how it’s kind of daunting to just let go.

Jules: Yeah. I think that I learned something in this process. Doing it on this weekly cadence meant that I had the week — it was sitting at the back of my head. I have this joke that I do my best work when I’m asleep. It meant I could spend the week just leaving the stuff boiling over, and then sit down, make changes to what I’d already done, and then carry on writing. And I didn’t waste any time because it had been mulling at the back of my head. I could just get the stuff down. And that was, I think, a useful thing to learn.

I got to the end and then I kind of got stuck because there were lots of things that I wanted to do but I couldn’t do. Things didn’t quite fit in the right order. I couldn’t get the pictures that I wanted. I’d got analogies that I wanted to put in but that didn’t seem to be gelling. And so I went through this process of getting a bit stuck and I had to start pulling it apart again.

But I thought, I really want an index, and indexing just scared me. Every time I tried to do it, I just didn’t know how to put an index together. And then a friend suggested I use Claude for it, and Claude did a smashing job — I mean, it didn’t give me everything that I wanted, but it did 90% of it, and then it gave me something that I could start pushing into place. And I was surprised at how effective that was. So that was another learning — that there are points at which I can trust AIs like Claude.

Len: Yeah, I mean, I could talk about that for a long time myself as well, with all the things we’ve done at LeanPub. One thing in particular — there are people you’ll talk to for whom the index is a form of art and they’re like, I would never hand that over to an AI or something like that. And there are other people who want an index and they’re like, I’ll review it and I’ll make sure it’s right in the end. But if I’ve got a 300, 400 page book and I can get something started for me to check basically, rather than just building from scratch — that’s a perfect example of that.

Jules: What I wanted to do with the index — I mean, I wanted some jokes and Easter eggs in there. That’s easier to do than it is to write the whole index.

Len: Yeah, no, 100%. And yeah, it’s interesting — you’re reminding me of my old approach to writing essays when I was a student, which was to write them the night before. But of course, I didn’t do all the work the night before. I did the work for weeks leading up to the moment when I actually sat down. But everything was in my head and kind of ready to go basically, except for the things that you discover along the way, which is the excitement of writing.

The last question I have — actually, no, there’s one more thing, because you mentioned it. So you did consider going with a conventional publisher, doing a publishing contract and stuff like that, but you decided not to because basically you were getting constraints placed on you for what you could do — that’s what I took away from what you said. But what form did it actually take? Did you write emails? Did you have phone conversations? How were you reaching out to and having these conversations with publishers?

Jules: Every publisher has got a process by which you can make an unsolicited approach. I found those absolutely useless. Everything just went into the void. It’s sad to say this, but every single conversation I ever started was as a result of a personal introduction. It’s not what you know, it’s who you know.

The people who I met were generally very decent, very honest people. But they were book factories. They were absolutely about shifting volumes. They wouldn’t do any promotion for you. They wanted a uniform style — they wanted a uniform cover style, but they also wanted a uniform writing style. And in the course of writing this, I remember giving a draft to a colleague. And he said to me, well, I really like what you’re saying, but I don’t like the style. If you want to be taken seriously as an academic, you’ve got to develop this academic writing style. And I said, what makes you think I want to be taken seriously as an academic? I want to be taken seriously as a practitioner. I’m showing off my war wounds here. That’s absolutely not what it’s about. That attitude was a very similar attitude from the publishers.

The other thing that the publisher was offering was rubbish returns. Most of them were saying that a really successful technical book will sell 3,000 copies and get about 50 pence from each one. So I’m looking at lifetime earnings of about 1,500 quid. And lifetime means lifetime. They never give it back. It’s never out of print. It’s always on electronic. And so even if it’s not doing anything, you can’t get it back. You can’t do anything else with it if you want to. You can’t even do a second edition.

And I really wasn’t comfortable with that. So for all of those reasons, I thought that LeanPub was a better bet. And incidentally, between starting it on LeanPub and now, which was only five years, I have made more money out of LeanPub than I would have done out of lifetime sales from these publishers. I mean, of course, it’s not just about the money, but it’s about money, right? And so LeanPub was offering me a much more personalized experience. And it was, in retrospect, a lot more money.

The downside is you have got to do it all yourself. I did not have a production team around me. It’s a lot of work. And I had no idea how much work it was going to be before I started. And I would offer that as a warning. It’s big. It only takes five months to make a baby.

Len: Yeah, I know. It’s interesting. One thing — just to pick one of the many things you said there that resonated with me — obviously, I work at LeanPub, so the authors I’m going to be talking to are going to be people who either have had experience with conventional publishing processes and didn’t like them, or realized that this project wasn’t for that. But one of the most consistent things I hear is, the people I interacted with were great, it was the processes that they were a part of that I didn’t want, that just didn’t work for me.

And then, of course, the other thing is money. Making 5% versus 80% on the sale of an e-book changes the math for what you do. And a lot of people, when they hear talk about money, think, oh, are you being greedy or something like that? It’s like, no, but I do have to justify the way I use my time. And one of the ways I justify that is, am I going to get — I don’t work for free. And basically an 80% versus, let’s say, a 5% or 10% royalty rate changes the value of a book project. And that’s one of the things we’re proudest of — that by providing a high royalty rate like that, some book projects, some books happen that otherwise wouldn’t happen at all because of that. And there are a couple of LeanPub authors with Lambos, as they say, that they never would have had if it weren’t for that kind of thing. So that’s just so great to hear.

Jules: In fact, one of the interesting things that happened with several publishers was they said, well, we don’t like this book because it doesn’t fit in our catalogue. But we’d like you to write this. And yes, sure — several times I did treatments on books. And eventually I thought, well, I believe in the book, but not in you. So these are sitting on my shelf waiting to be written.

Len: Well, I look forward to reading them one day. I hope maybe you need a commuting job again, if you don’t have one right now, in order to do that. But I look forward to your future books as well.

So the last question I have for guests on the podcast who’ve written a book on LeanPub is always: if there was one magical feature we could build for you, or if there was one thing that whenever you’re working on LeanPub you have you shaking your fist at your screen, shouting at your monitor going, damn you LeanPub — if there was one thing we could fix for you, can you think of anything you would ask us to do?

Jules: I find that really hard, actually. First of all, because I’ve had what I think is a very successful experience on LeanPub. There’s not very much I would criticize or complain about. But on the other hand, I haven’t used a great deal of the platform. I’ve used the publishing part, but I haven’t used the markup language because it didn’t fit the book that I wanted to write. So it’s hard to answer. I think if I was going to ask for one thing, it would be LaTeX integration in the way that you’ve got the other integrations ready.

I understand this is a really complicated thing to deliver, but for example, I’ve now got it in a print edition — it launched a couple of weeks ago, so that’s now on Amazon. It was very easy to get from that into a print book, but getting from what I’ve got now into, say, Kindle — getting it into a pure electronic form, that’s going to be a lot harder. Now, that’s something which you do really, really well. But again, I can’t use it because I’m not using your tool chain. Having the LaTeX integration would really help that. And in saying that, I know that’s a really big ask.

Len: Yeah, no, that’s very interesting. I will convey that to the team. I mean, if you do write in Markua, our markup language, if you write your book in plain text, that way we have all kinds of amazing things that drop out of it, including a print-ready PDF that you can just click a button and get, which is what you need to go to Amazon or IngramSpark or other print-on-demand distribution things. And of course, we have an upload writing mode, which you would have used, because there are a lot of authors who are like, ah, I’ve already written a book but I don’t want to have to learn anything new or reproduce my entire manuscript. Or other people are like, I really prefer this other way of doing things, I’ve got my own way. So we do end up with people who show up having used their own tool to produce their e-book files and they’re publishing them on LeanPub with great success. And they’re like, ah, but there are features that you have that I’d like to use that I can’t. If only you could convert my arbitrary format into yours — that would be amazing.

And that is actually something that we are thinking about and working on. Hopefully the day will come when we could take a manuscript like yours in its raw format and turn it into whatever is required for the next step or the next part of the process. That would be something that we’d love to be able to do. And it’s definitely on our radar. But as you expressed, that is a lot to do.

Jules: And I accept that. So I accept that I have to do a chunk of the work itself. And I do know that for the next — I’ve got two in mind, and there’s one which is the next thing I’ve got to do — that is also going to be LaTeX.

Len: Okay, well, Jules, thank you so much for taking the time to talk to all of us today about your book, Extreme Reliability: Programming Like Your Life Depends On It, and about your background and career, and how software really works and should work, and in particular your experience designing airplane software and stuff like that. I really, really appreciated your knowledge and experience about that. So thank you very much for being a LeanPub author and for being on the LeanPub podcast today.

Jules: And thank you very much for having me on.