Dave Farley, Author of Continuous Delivery Pipelines: How to Build Better Software Faster

Len: Hi I’m Len Epp from Leanpub, and in this episode of the Frontmatter podcast I’ll be interviewing Dave Farley.

Based in Tring, northwest of London, Dave is a consulting software engineer, pioneer of the concept of Continuous Delivery, and a popular software development thought leader and speaker.

Dave is perhaps best known as co-author of the Jolt-award winning book Continuous Delivery, and more recently, Modern Software Engineering, as well as being the host of the incredibly popular YouTube channel Continuous Delivery.

You can follow him on Twitter @davefarley77 and check out his website at continuous-delivery.co.uk, and read his blog at davefarley.net.

Dave is the author of the Leanpub book Continuous Delivery Pipelines: How to Build Better Software Faster.

In the book, Dave helps readers understand how to build software using the Continuous Delivery Deployment Pipeline, and the principles underlying the best state-of-the-art process there to get from an idea to working software.

In this interview, we’re going to talk about Dave’s background and career, professional interests, his book, and at the end we’ll talk about his experience using Leanpub to self-publish his book.

So, thank you Dave for being on the Leanpub Frontmatter Podcast.

Dave: It’s a pleasure. Thank you for inviting me.

Len: I always like to start these interviews by asking people for their origin story. So I was wondering if you could talk a little bit about - going way back to where you grew up, and how you first became interested in computers and technology?

Dave: Yeah, sure. So my first ever computer, I used a computer at school. And in those days, it was a dumb terminal connected to a mainframe in a university. This was kind of pre the birth of microcomputers and home computers. I wasn’t wildly impressed at that point. I remember being impressed that one of my friends managed to make it say rude words whenever you logged in, but that was all.

Later, I was living with my girlfriend at the time, who’s now my wife, and has been for a long time. Her father worked in a university. He came home with a computer over the summer holidays, which we borrowed. It was just a ZX81, a little Z80 based microprocessor in a box with a touch sensitive keyboard, with 1K of RAM. I got interested. So my wife taught me to write my first program. Then I got really interested, and started writing games for it, and learning assembler, and all those sorts of things. Then went on from there.

My first professional job was as a support engineer for a computer manufacturer answering people’s problems, and trying to help them use the computers. Then I got a job in an R&D company for another computer manufacturer, doing system software, and all that kind of stuff. Got interested in more distributed computing and bigger scale systems, and ended up building some interestingly complicated systems.

During the course of my career, I was involved in a delightful project, towards the end of my coding career. Where I led a team that built one of, if not the world’s highest performance financial exchange, from scratch, using the practices of continuous delivery. Which was a remarkably fun experience, but also very challenging.

Len: I’m really interested in actually talking to you a little bit about the details about that. I was sort of, not in trading, but I’m a former, for my sins, a former investment banker.

Dave: Oh right. Okay, cool. So we were doing a really interesting thing. The company was called “LMAX.” Still is called LMAX. We were doing an interesting thing. Because it began as a spin off from a sports betting company, called “Betfair.”

Betfair were trying to do the same kind of thing for financial trading, that they’d done for sports betting. Which was kind of democratize it. Give it open access to the markets. The vision for LMAX originally, was that anybody could kind of turn up with $10, 10 pounds, whatever, and trade on an equal basis, with - in a big institutional - investment banks.

We built a system for that. But that was an - What was interesting technically for that, was that it kind of combined two very difficult problems. Low latency trading is about doing very, very efficient algorithms, and sort of the narrowing down of performance and speed to hit the price in a market, within microseconds. That’s a tough problem. That’s a world class problem right there. But usually in those sorts of exchanges, the number of players in an exchange like that, are usually measured in the tens. It’s not thousands or millions of people.

But we wanted to democratize it. So our vision was that, one day, potentially millions of people might rock up and be trading in this. So we were also doing the internet scale thing. So we got these two horribly complicated issues of performance and scalability. They intersected where the prices met. So we did some fairly innovative things there, and we came up with some smart ways, if I may say so myself, of doing a variety of things to get the performance out of that system. So it was really interesting.

The other thing that was really interesting, from my point of view, was, at the time, I was in the midst of writing my first book, which was the Continuous Delivery book. And, so, I was hired as the head of engineering for that company. So, given the task of creating the team, and setting the direction that we go in terms of development process.

So we started practicing the continuous delivery approach to software development, from day one, with a blank sheet of paper, in this incredibly challenging problem domain. So, it was like a green field, experimenting, what’s possible if you do continuous delivery with hard problems. So I learned a huge amount from doing that.

Len: Yeah, it’s really interesting. I’d like to talk to you more specifically about continuous delivery and that. But I wanted to sort of go into a little bit of the details about LMAX there. Because it’s such a - it is such a fascinating problem. I love the fact that it, sort of, it’s origin is in the freedom that Brits have always had to bet.

Dave: Yes.

Len: And, I think, I’ve interviewed at least one or two people who became sort of like chief software architect types later on in their careers, after working on these early on kind of Betfair-type companies. Because it was just like, you could’ve come from anywhere in IT, and suddenly you find yourself in these hot houses.

Dave: Yes.

Len: Of growth, and sort of thriving in ideas, to try and take on these challenges.

So people understand a little bit about how this kind of things works, like you’ve got to get - there’s sort of like people buying, and there’s people selling, so there’s bid and ask prices. There’s got to be a single price that people are kind of bidding on. This is all happening like 1,000 times a second, with millions of people.

Dave: Faster than that in our exchange, so -

Len: Wow.

Dave: So we set ourself the goal originally of turning it round in a millisecond, 1,000 times a second. Ended up outperforming that. Because one of the things that we learned, was that that wasn’t quite quick enough. Because you had bursts of performance. So 1,000 times a second was the slowest that we could go.

Len: Wow.

Dave: So we had to come up with bursts of performance that were faster than that. You ended up - it was about - for the technical, any technical people, it’s about 80 microseconds round trip from hitting the edge of our network, going in, performing a trade, and giving an answer. Which is phenomenally fast. At that point in history.

Len: Was this the kind of thing that the rest of us would’ve read about in The Economist, about how like trading outfits are sort of setting up their outfit to be closer, because there’s like the limitations of the speed of light, and stuff like that?

Dave: Yeah, yeah. So I did some of that before, but absolutely. So we were - in the trade, they call them “colo’s,” co-located trading systems. So literally, as you say, to minimize the speed of light costs, people will put the algorithms that are performing the trades, in the same data center as the exchange. So that they’re minimizing the travel through wires, because that’s going to slow things down.

It’s an interesting problem. Because the way that you described it is nice. So you’re trying to hit this price, and what’s - the reason for the performance, I always found really interesting. Because it’s a real business driver for those performance needs.

There are organizations that are called “market makers,” who place trades in exchanges. The job of a market maker is to keep the stocks liquid in the exchange, in essence. There’s usually somewhere, that, someone that you can buy off, or sell to in the exchange. The market makers fulfil some of that purpose.

Market makers make money on the spread between the buy and the sell price. So it’s a bit like, if you go to your - if you go and buy money, foreign currency, to go on holiday, you spend a bit - you buy it for a bit higher price, and you sell it for a bit lower price. There’s a difference in the price.

So market makers make money on the spread between the two prices. So what they’re trying to - they’re competing with one another. What they want to do, is they want to have the narrowest spread, so that they’re most likely to get picked when somebody places a trade in the exchange. So what they’re trying to do, is they’re trying to identify, “What’s the best price right now? I’m going to put the best price right now. Then a few milliseconds later, I’m going to pull it, and I’m going to replace it with my new best guess at a price.” So what they need is this consistent latency, the turnaround time of being able to place the price and get an answer back.

The trouble is, is that time, as it expands out, goes wider and wider. So the risk gets bigger. That means that the spread gets bigger. So the way that they keep the spreads tight, and win, is by being very efficient, and not going too far forward on the time horizon.

I don’t know whether this is too technical? But it always interested me, that idea of the time cone sort of expanding, and trying to work within this narrow time window that’s available for the trading.

Len: It’s interesting, actually, mentioning sort of like light cones, and stuff like that. You’ve just reminded me of something. That’s very interesting.

So with the introduction of systems like this, when one thinks about stock trading, sort of from the movies, you might think of all the way back to sort of a guy in a top hat, looking at a table on the - in the newspaper, or something like that. Then you end up with Humphrey Bogart in the back of a limousine, with a kind of Teletype or whatever - well no, not Teletype. But anyway, some kind of like early mobile mechanism. But then you get the like trading floor, people yelling at each other. Then you get the sort of late 90s after the web was introduced, and there was sort of - people could do trading like from their computers, and things like that.

But when stuff like what you’re describing got developed, then things could start happening, where you could start applying sort of scientific principles. I’m thinking specifically about a friend of mine, who, he did his doctorate in atmospheric physics. He was trying to figure out how to get satellites to look through clouds better, when they’re pointing at the earth. He became a quant.

For anyone listening, a quant is someone who sort of takes all of this, these wonderful, kind of, fascinating kind of mechanisms, like LMAX built.

Dave: Yeah.

Len: So it’s not necessarily a person sort of typing in bids and asks, or clicking buttons. But it’s actually algorithms. These really complex algorithms that are carrying out these trades.

Dave: Yeah. After LMAX, my last job as an employee, before I started my consultancy dealing with software engineering - but I worked for a trading company that was on the other side of the fence. We were trading in exchanges. We were doing the market making thing that I just described. Quite a lot of the quants were scientists or mathematicians of different flavors. It’s, if you’ll forgive me being flippant, it’s quite a good way to earn money, if you have that kind of mind, of being a quant.

Len: No, no, definitely, definitely. And, but as long - but there needs to be - the interesting thing I’m very interested in about it, is there has to be a system in place, to which these scientific kind of principles can be applied.

Dave: Yes.

Len: That’s something, that, in many different ways, software brings to the things we do in the world, in business and things like that. This is something that you write and talk about quite a bit. In particular, for example, continuous delivery is the application of scientific - the way you define software engineering, I’m quoting yourself back at you here. But, “I define software engineering as,” quote, “the application of an empirical scientific approach to finding efficient solutions to practical problems in software.”

Dave: Yes.

Len: What’s fascinating about that, is that the word “software” could be substituted in that sentence for lots of other different things.

Dave: Yes.

Len: Then you have some other kind of engineering.

Dave: Yes, absolutely. I’ve made this joke before, but I started my career with job titles like “junior software engineer,” or something like this. Did nothing that was even vaguely like engineering. Then in the middle of my career, I worked at places that really didn’t like calling what we do “software engineering.” Then, now, after my experience building exchanges and doing trading systems, and stuff like that, I think it absolutely is engineering. When we apply engineering thinking, it really matters. It makes a huge difference to the effectiveness of what we do.

I’m a science hobbyist. I’m not - I did O levels and A levels in science subjects. But I love science, and I read about physics all the time, and that sort of thing. Science is just the best, humanity’s best problem solving technique.

So if you’re a software developer, or software engineer, and you’re not interested in science, I think you’re missing a big trick. Because our job is all about problem solving. And, so, absolutely, we should be applying science, scientific ideas. Not deeply formal, not sigma proofs of things, but certainly applying a science style approach to problem solving. I think that’s a really good description of engineering at all, and practical use of scientific principals to solve problems.

I would agree with you, it’s - it works better than other things. So I’m a human being, so I’m not naturally, I don’t apply science all of the time when I’m choosing what I’m going to have for dinner, or anything. But I think, when you’ve got problems to solve, and difficult problems, that’s how we do it, that’s how human human beings do the best job of solving problems.

So, I’m quite passionate about that. Because, for people that aren’t familiar with software, strangely, an awful lot of software development is not like that. It isn’t practicing those. It’s quite often based on guesses and hunches.

Len: Yeah. That’s a really fascinating thing, that I’d actually like to go into a little bit deeper about. About the fact that software development is actually very creative.

Dave: Yes.

Len: A lot of people have the idea, that like - and, like, a lot of people who are behind decisions that go in - that ultimately end up in software being created, also have a mistaken belief about what software development is, and what programmers do.

Dave: Yes.

Len: But just before we get to that, actually, so you brought up O levels and A levels. For people not from the UK, these are sort of like things you - levels of education that you take.

Dave: Yes.

Len: At the end of your sort of public education. There’s a - so there’s something, that’s - I’m curious to talk to you about, because it’s come up many times on this podcast in the past. From people, from, people from different generations. If they started in the age when you sort of like had computers with 1K of RAM.

How you learned things differs from when you’re in - like nowadays, when there’s just incredible amounts of stuff available. I guess the question I have, is, if you - I mean, you’ve had such an interesting career. But if you were starting out now, and you were 18, would you get a four year Computer Science degree, if you had the intention of going into a career in tech?

Dave: I think, no. You mentioned my YouTube channel earlier. I’ve done a few videos that are aimed at junior programmers entering the industry. I’ve thought about this quite a lot. Because, I, as you say, when I started, it was very different. I think, my impression, as an employer of junior developers in the past, and observing it from a distance, and talking to people. My impression is, that it’s still incredibly difficult to get your first job. So there’s a huge demand for software developers.

But what nearly all employers want, is that they want experienced people, who can kind of come in, and do what they’re already doing. I think that’s a mistake. I think that’s dumb on the part of the employers. But I think it’s the way that it works. So, there’s this big barrier to getting in.

Then once you’re - once you’ve got your first job, it’s an awful lot easier then, once you’ve got a little bit of experience, to get subsequent jobs. One of the barriers to getting in, is to have a degree. That makes it much easier to - not much easier, it makes it easier. It’s still tough.

So, I don’t think it’s wrong, I don’t think it’s enough. The - I don’t think that, mostly, of course, there are good schools, and so on. I don’t think mostly software engineering courses, Computer Science courses, and so on, I’m honest, are teaching the right things, from a practical, pragmatic, professional developer point of view. You get a young person coming straight out of university, they’re not ready to do professional software development.

Now, that’s okay. That would be the same if you said, a new surgeon coming out, or they’re going to need some support, and they’re going to need some on the job training. So, that’s okay.

But often, in the past, I haven’t seen a very big difference between people that have done a software degree, Computer Science, software engineering, that kind of stuff. People that have done other technical degrees. So I’ve hired people with chemistry or physics. Actually, they were usually better at the problem solving stuff. Which I think is the core of our discipline. Our discipline’s rather strange, and a little bit - partly because of being new, I think. I think you -

Our job as software developers is not to write software, that’s the tool that we use to solve problems.

We’re still, although we’re 50 or 60 years old as a career path, we’re still sort of finding our feet, to some degree. I think that we - one of the mistakes that we make, is we focus too much on the tech, the shiny things. Because that’s one of the things that appeals to us as technologists, and to all of us. Not enough on the problems that we solve, and the value that we bring. Our job as software developers is not to write software, that’s the tool that we use to solve problems. Our job is to solve problems. I don’t think that higher education, in the round, tends to teach that to software developers. I think that’s a mistake. I think that’s -

They should be learning other things. More fundamental things. So in some ways, less complicated things, but more valuable, more durable things. I think that they tend to focus on some of the wrong stuff.

So I don’t think you’re going - By doing a computing degree of some kind, I don’t think you’re going to learn all that you need to be a good software developer. But it’s a useful tick in the box, for getting that first job.

The, my advice to anybody that is really interested in starting this career, this as a career - it’s, I still think, I still believe it’s a wonderful career. It’s hugely creative, very challenging, lovely problems to solve, and it pays quite well. It’s not a bad kind of thing to do for a living at all.

But I think that, if you want to be good at it, and why not? Why wouldn’t you want to be good at it? You’ve got to do a lot. You’ve got to write a lot of code. So part of the game, I think, is playing with code. Just do fun things. Just write code, and it doesn’t matter what it is. Doesn’t matter what you’re doing. Just write code for fun.

I spent, I described the start, my start, and I spent quite a lot of time just writing games for myself, initially. Learning computer programming by writing games. Then I got interested in computer graphics, and doing some more sophisticated things with computer graphics. I built ray tracers and 3D graphics engines, and all that. All on really crude ancient computers, by today’s terms. From scratch. I enjoyed that a huge amount. But learned such a lot, just by playing with the technology and learning it.

But my friend who I worked with at LMAX, who was, he was the CTO, I was the head of software development. We used to do all of the interviewing for the technical staff that were coming in together, as the final interview. Our shorthand for what we were looking for in a good - this will date both of us a little bit. But our shorthand was, that we were looking for the people, that, when they first saw a video recorder, they took it apart. You want that curiosity to understand how things work, and to play with them to figure out how they work. I think that’s a huge part of the, what differentiates, what I perceive as being the great, great software developers that I’ve worked with. Versus the more ordinary ones.

Len: Yeah, that’s a really - thank you very much for that really great answer. I mean, the idea that like, the sort of formal education can sort of help you get a foot in the door, is just true. We all know that. But get your hands on it.

Dave: Yeah.

Len: Take it apart, build it, is a really great way to sort of like learn the problem solving things, just sort of like directly.

But you mentioned a couple of very important words there, one of which was “agile,” and the other one was - tou used the word “durable” actually. I had this in my notes from an interview you did, for, I think, called a sort of series called, “The Engineering Room” on YouTube -

Dave: Yeah.

Len: With Martin Fowler. You had this line, where you talked about durable ideas. I was looking forward in this interview, to ask you about that.

So you just now mentioned this idea that, particularly when it comes to sort of software, we’re sort of still at the beginning, even though we’re 50 or 60 years on.

Dave: Yeah.

Len: That reminded me, I had the pleasure of interviewing Jerry Weinberg a couple of years ago, before he passed away. He had this really great story, that I hope I don’t mangle too much, about being at IBM, in the early days when it got into computing. Being in a room with a bunch of executives, who were just terribly disappointed by the idea that they’d have to have programmers.

Dave: Yes.

Len: That writing software was just going to be part of their business going forward. They sort of came into the notion, that like, it’s kind of like, “We’ll get the spec, right? We’ll get the spec for this programming stuff, then we’ll be done with that, and we’ll move onto our machines.”

Len: When they learned that that sort of programming was going to be something, they were disappointed. This, like this - I just love that story. Because there’s something about older management practices, that just doesn’t map onto the work that programming entails, right? So, for example - my brother’s - Who’s an English professor, has a great joke, that like, I’m just going to put my chin on my hand. “When you see someone doing that, they might be doing the hardest work you’ve ever seen anyone do in their life.” But you can’t see it.

Dave: Yeah.

Len: You can’t see the work. I’ve always thought that this is the - this, at least, this is my sort untutored opinion. Is that, this is actually part of the problem. So what the people want is like, they want to measure lines of code. Or at least, “I want to see a number for lines of code. I want to see people - I don’t want -“ Like, “I want to take away Facebook,” that’s an old reference now, I suppose? But, “Take away Hacker News from them. I want to take -“ There were times when sort of people wanted to take away, like, basically the web from workers who were at computers, because they didn’t want them distracted. It’s kind of like, “Well that person being on like some Reddit thread, might be actually like contributing to a state of the art conversation about his job, and, or learning something state of the art from there.” Just with that -

Dave: Or answering a deep question, that otherwise they would spend weeks trying to solve by doing it alone. Absolutely.

Len: Yeah.

Dave: Sadly, I don’t think that’s necessarily gone away. I think that is a disconnect that still often exists. It takes different forms. But in my - I spend most of my career either making YouTube videos, writing books, and those sort of things, or consulting with big companies, to help them improve their software engineering. I still see facets of that all of the time.

I think you’re right. Software is interesting. It’s strangely difficult. At the same time, strangely approachable. It’s easy to get started. It’s really easy to write your first program. I can show you how to write your first programming, under 5 minutes, and it’s simple. The trouble - you can learn the basics in an hour or two, easily, just to write simple programs. They’re not going to do very much. You’ll need to learn an awful lot more.

But it’s kind of - it’s going to take you a lifetime to get really good at it. There are aspects of software development, where you’re a tiny step away from some seriously deeply complicated problems. The two biggies are concurrency and information sharing. So concurrency is when you’ve got more than one thing going on at the same time, and we’re just not very good at thinking like that, even though that’s the way that the world works. So when a software - the way that a programmer learns to write a program, is, they issue an instruction, and they wait for an answer back from something else, in response to that instruction.

That’s quite a - when you stop and think about that, that’s quite a weird thing. You and I having a conversation, but while I’m talking and you’re listening, your brain hasn’t stopped. You’re still thinking about all sorts of other things, and they’re all going on in parallel with me talking. That’s the way that the world works. That’s the way that the universe works.

But not in software. When you first learn, you learn this kind of synchronous series of steps. “You do this thing, then you do that thing, then you do this other thing.” Except, that’s not the way the world works.

As soon as you get into concurrency, you’re exposed to all of this other complexity. You’re doing this thing. Something else is going on over here. Depending on when you finish doing that thing, or when it tries to talk to you, you’re in a different stage. So that’s complicated. That’s hard to think about.

World class concurrency experts say, “Try to avoid doing concurrency, because it’s that hard.” But it’s really, really easy to do in software. So everybody does it, and so they trip themselves up, and write things that are complicated, as a result. Then there’s information.

You’ve got two pieces of information, two copies of information, and it doesn’t matter what the nature of the technology. If you and I have got two pieces of paper with the same information on them, and we’re both independently changing them, which one’s true? Which one’s correct? How do we bring them together again and make them correct?

Google Docs kind of makes this live. You can edit - which would you prefer? Would you prefer to be doing an editing job with me, writing a book, and writing it on pieces of paper, and exchanging these pieces of paper, or sharing it on the screen with you, Google Docs? While you can see what I’m typing, while I’m typing it. You can be typing somewhere else in the doc. That’s a much more effective way of interacting, that technology enables.

But that’s a visible showing of the problems of managing data in multiple places. Computers service that in a way that’s unusual and complicated. It’s very, very easy to step into that level of complexity. Which I think is world class. I think, to do a good job of that at scale, requires deep thought to achieve it.

Len: There’s another, you talk about, somewhere, I think on your website? About how the most fulfilling lines of code you write, are those that you first show someone what it can do, and they’re like, “Oh my God, that’s amazing.” Then you show them the code, and they’re like, “That’s it?”

Dave: Yeah.

Len: But this, again, so when you’ve mastered these problems, and when you’ve built solutions to them, and then you show someone who’s not a programmer, and who’s maybe an executive, or something like that. There’s this, just, it’s how do you conceive of the value in that? You know what I mean? It’s just -

Dave: Yes. I mean, it’s incredibly difficult. I think that - our industry’s been kind of polluted, for want of a better word. Because when we’re talking about physical things, the production of physical things is a really difficult, complicated, hard problem. So nearly everybody that doesn’t write software, thinks in those sorts of terms about it. So big organizations, they’ll want to scale it up by adding more people. So, divide it up into steps, so that each person’s focused on one part of - that’s almost disastrous to do that.

There was some lovely piece of research that measured the productivity of teams. They, it was a metadata study. They divided the cohort into teams with five people and fewer, and teams with 20 people and more. Then they just kind of counted how much code they wrote. Just how many lines of code. How long it took them to get to 100,000 lines of code. You’ve already said it’s a really rotten measure. But it’s something that you can do in a metadata study.

They did that, and the teams of 20 people or more beat the teams of five people or fewer, as you might expect, to producing 100,000 lines of code. Except, on average, for all of the teams, it took nine months. The teams of 20 people or more beat the teams of five by one week over a nine month period. So in effect, a team of five people is nearly, but not quite, four times as productive as a team of 20 people, when writing code. That’s just in terms of the amount of code. Then you look at the number of errors and mistakes in it. The teams of five people kill it.

So what you want, is that you want to organize software development into many small teams. That’s complicated. That really hurts people’s heads when they start thinking about it. Because they’re used to production lines, where you just have armies of people doing simple tasks. You want small teams of people doing complicated things. Not many people doing simple things, because that doesn’t stack up in software development, because it’s a bit too complicated for that. There’s lots of counterintuitive things. It’s a reasonable assumption.

But in software, we don’t have a production problem, because we can recreate any system that we make for free. Or close to it. So that’s not our problem. Our problem is only about design. So it’s a bit like saying, “I want to paint the Mona Lisa, so let’s hire an army of people and tell them that we want the Mona Lisa, and they can each do a little dot piece.” That’s probably not going to work very well. I’m not saying the stuff that I’ve done or other people do is the Mona Lisa. But just as an analogy. It’s - you’ve got to - it’s more complicated than it looks on the surface, I think? We’ve taken quite a few missteps.

Which is really a - I think of my own work, that’s a lot about, of what it’s about, is trying to help people see the wood for trees, and focus on what really works. I think we know what works. I think there’s a great consensus in, certainly the experts that I know in the industry, there’s a huge degree of agreement to that, what works. It seems to me.

Len: Yeah, and actually on that -

Dave: But it’s still rarely practiced.

Len: Yeah, that, actually, maybe this will be connected to something that I’m going to mention now? So it’s interesting, in the history of sort of software sort of development, and IT and computers and stuff, there’s - there is a sort of thread of understanding that maybe older management practices, and business practices that came from sort of physical production lines, held back kind of software development practices and techniques, and stuff like that.

But it appears we’re now - and, again, I say this just as a kind of headline reading, observer of things, right? But we’re maybe in a point where sort of like software development practices are now improving production line practices. And, so, for example, one example you brought up, I think a few times? Is Tesla managed to, in one of its factories, switch its model three production to sort of have like higher storage capacity. But they did this in three hours.

Dave: Yes. Which also involved reconfiguring the factory to produce new, the different design of the car, in three hours, from the commit of a piece of software, running through a bunch of tests, having it deployed into the factory. After three hours, the factory was producing new cars to the new design. All the time while not losing the production of the old cars, until it was ready to do the new ones. Is just phenomenal.

I think you’re right. I think that one of the things that - there’s this interesting dance, for, if you’re a process nerd like me, there’s this kind of interesting dance around the way in which we’ve evolved. So we started off - it started off, everybody ignored software development. In the sixties and so on, they just - it was - there was a lot of freedom for people to do things, and there was not much management. Then people saw that software was getting important, and worth money, is valuable. So, started to apply some more management techniques, but took the misstep of trying to apply production line style thinking, because that’s what everybody’s used to.

I think that you’re right. That if you look at the leading edge companies these days, it’s kind of gone the other way. So software and the scale of software that we’ve been doing things, has challenged conventional thinking, in lots of different places, but certainly management thinking. That’s influenced the way that successful organizations are structured, and work, and so on.

The classic elemental model of the Silicon Valley industry, was kind of given birth by the electronics companies in the 1950s and early 60s. They had a different way of going about it. Because they were doing some genuinely hard stuff. That’s kind of continued, I think?

I think SpaceX and Tesla are fantastic visible examples of that transition. Both SpaceX and Tesla are continuous delivery companies. They apply the approach that I describe and recommend, but kind of across, the way that their factory works. That, I’m a nerd, and I’m busy watching SpaceX build their Starship to go to Mars at the moment. You can kind of watch live, cutting edge engineering on YouTube, and get updates every week, kind of thing on what’s going on. That’s very, very obviously based on the same kind of principles that I talk about, when I talk about software engineering.

It’s about being iterative and experimental and empirical, and gathering feedback and learning and evolving the systems, and then building systems that are modular, and have loose coupling and good separation of concerns. They’re changing the design of these massively complicated, massively massive spacecraft, every single day, all of the time, it’s a complete, continuous evolution.

That’s really at the heart of software. I think that software liberated that kind of thinking. Because software’s so malleable. But now, it’s going back, and it’s being applied in places that aren’t so malleable, but still to great effect.

Len: Yeah, that’s a wonderful example. I mean, the idea - I think that, again, that you’ve talked about a few times in sort of podcasts and stuff that I listened to, preparing for this interview. Is the, like SpaceX will update the software on a rocket 45 minutes before a launch.

Dave: Yeah.

Len: Which, from what I understand of the, as it were, legacy space industry, is insane.

Dave: Yes.

Len: You have to have a process that is sort of self-reinforcing, in order to trust. Like it has to be built into the process, that you can trust it, so that you can -

Dave: Yes.

Len: This is where like, always. So this is, and this is me, sort of priming you to talk about continuous delivery. Maybe to people who’ve never heard about it before. We’ve said, it’s not just about software. It can be applied to manufacturing and rocket ships, and cars, and things like that, as well. But the idea that you’re always ready to deploy.

Dave: Yes. Exactly.

o it’s really - the way that I think of continuous delivery. So my favorite way of describing it, is working so the software, products in general, I suppose, but certainly our software is always in a releasable state. What that is really, is kind of this amalgamation of kind of, the engineering and science stuff that we need to do, to be able to build good products. But brought together with sort of lean manufacturing thinking. So we’re trying to work in it as efficiently as we possibly can. To take as much work out of the process as we can, so we can work efficiently.

One of the things that that demands of us, is to not create waste by building bad products, whatever they may be. So one of the sort of total quality thinking sort of ideas that we apply in that sense, is this idea of working so our software’s always releasable. So we’re going to make a small change, and validate it, very quickly, very efficiently, with high levels of automation, throwing computers at that, solving that problem. Validate it very quickly, to just check, “Does it still do what I thought it was doing before? Does it do -? Is the code that I wrote doing the things that I think it’s doing?” Give us that feedback very, very quickly.

That’s where, ultimately, Tesla being able to make a change to a car and reconfigure the factory in three hours, comes from. Because it’s all hugely automated. You make a change, and that change goes in, and runs a bunch of tests to validate the change. Then it goes into, it runs another bunch of tests and simulation, to validate that all of the pieces work together. Then it burns the silicone, and then it updates the factory or the car, or whatever else it is that the software is for. That’s the kind of idea. So we’re working with these very, very fast feedback loops, and our objective is to make progress in tiny little changes. One of the -

One of the outcomes of working in these smaller steps, is, it sounds horribly risky making these changes really quickly. Actually, it’s safer.

I think that I, I think this is fair to say? That it surprised most people. Certainly surprised me. One of the outcomes of working in these smaller steps, is, it sounds horribly risky making these changes really quickly. Actually, it’s safer. Because each of the steps is simpler and less risky.

If it does go wrong, it’s easier to back out from the change, and correct it. It’s like - we’re going back to SpaceX building their rockets. The system’s modular. So if they don’t like something, they can kind of take it off, and they’ll reconfigure this version of the rocket and change it. Then update what’s going to come in future versions, and all of those kinds of ideas.

So being able to work quickly in these smaller steps, is increasingly being seen as the safer approach. I’ve worked with companies building medical devices that can hurt people if they go wrong. We’ve been starting to apply this kind of idea to building those sorts of devices. As well as sort of more run of the mill, finance systems or ones - finance systems are high consequence systems, but the consequences are only money, not people’s lives, or health kind of thing.

So I think it’s interesting, the way in which - as you say, software, ideas from software are, to some extent, feeding into society and changing the way that we think about the way stuff works. A bit like the way that lean manufacturing did, and revolutionized manufacturing. I think the same thing’s going on, to some degree, in other - the more creative parts of engineering as well.

Len: Yeah, it’s interesting to get into the sort of like gritty detail of what we’re talking about here. So a lot of people who do sort of programming, might be familiar with the idea of going off on a branch, if you’re building a new feature, right? This, I gather, comes, kind of more or less - modern iteration of this might come from sort of open source software development, right?

Dave: Yes.

Len: Where you’ve got lots of people working on a project. You sort of, you’ve got the sort of main trunk, as it were, of code. Then people make a branch off of it. I mean, this is a sort of crude image. But they - then - so basically, what they do is, they take it, and then they start adding or modifying something to add a new feature. Then when they’re done, they submit their changes for review. Those go back into the main. If they’re accepted, they go into the main sort of code base.

But with continuous delivery, it doesn’t work that way, right? You’re actually always working on the main branch.

Dave: Yes. So it’s about visibility of changes. So, and that promotes this small batch size idea as well, which we’ve been talking about. But primarily, it’s about the visibility of changes.

This kind of comes, continuous delivery is sort of the offspring of a practice called, “continuous integration,” which was made well known in the late 90s. The idea of continuous integration, was, if you’re in a team of people writing software - one of the other aspects of software that’s tricky for people that don’t do it - when we were talking earlier about some of the tricky aspects.

Is that how fragile it is. It’s intensely fragile. Much more fragile than words in a regular book, for example. So there’s lots of analogies that you could make with writing, for writing software. It needs to communicate ideas. It’s - there are people with good style and poor style, and all of those kinds of things.

But one of the differences is, it also has to execute it. It has to be readable by human beings. Which is very, very important. It also has to be able to actually work, tell the computers what to do, in a way that gets some outcome that we’re interested in. So it has to work. A simple, tiny error can cause your spaceship to blow up. That famously happened with Ariane 5. So, as a result of a software defect.

So it’s very fragile. So, it’s very easy, if you and I are both working on software, remember thinking about to versions of the information in different places. For us to make a mistake that’s subtle, and we - when we bring out changes together, we miss it, if we’re bringing lots of changes together at once.

One of the ways to alleviate that, is to just get better visibility, right? We were talking about working in Google. Wouldn’t it be great, if when I made a change to my software, you saw it almost immediately, and could understand the impact on the work that you’re doing, of the stuff that I’m doing? That would give us a better feedback cycle.

That’s kind of the idea behind continuous integration. We’re going to organize our work, so that we can get as close as possible to that. By definition, what continuous integration says, is if we’re not all looking at everybody else’s software, we’re not bringing it all together and continuously integrating it together to check that it all works at least once a day, it’s not continuous integration. So that’s kind of the starting point.

Then, branching, as you said, the most popular form of branching these days, is something called “feature branching.” It’s what, still what most teams do, I would guess? The way that people organize work on software development teams, is that somebody says, “Here’s a chunk of work for somebody to do.” The people pick up the chunk of work, they go away and work on it, on a branch, for that feature. When they think that feature’s finished, they’ll then integrate it with all of the rest of the software, to see if it works.

So now you’ve got a bigger chunk of software that might’ve broken something on day one, but took three weeks to write. You’ve got to unpick that. Meanwhile, I’ve got another big chunk of software, which made some different assumptions about the shared code. We’re trying to bring them together, and they don’t fit together. It adds quite a lot of work doing that, potentially.

But it was designed for open source software, and it’s fantastic for that. The feature branching idea with pull requests and all that, is fantastic for that way of working. But if you’re working on a team together, it’s not quite so good. It’s not - it brings in a lot of inefficiencies.

So continuous integration is this idea of eliminating some of those inefficiencies. So we can work much more quickly, much more closely together, by evaluating, to understand, working in smaller steps, and understanding the impact of our small steps by integrating the software together all of the time. Continuous delivery says, “We’re working so our software’s always releasable.”

So that means, that if we’re working using continuous integration, which it’s hard to do continuous - you can’t do continuous delivery, without continuous integration, really. If you’re doing that, then that means that after every tiny commit, that might not yet add up to a whole feature, I’ve got to be willing for that to go into production.

So that’s a big challenge. It changes the way that we think about work. It means that we work much more incrementally, and to some degree, more defensively. But it brings so many benefits, that it’s worth some of those costs.

There’s data that backs this up. So the data says that if you practice continuous delivery and continuous integration in the way that I described, then you produce higher quality software more quickly, compared to doing the feature branching thing.

Now there’s lots of people that do feature branching in lots of different teams, and they don’t like that answer very much, but that’s what - the closest that we’ve got to a scientific study of software development says. It says that this approach is more efficient. It does produce higher quality software, as far as I can see, as well as what the data says.

Len: Yeah, it’s really interesting you say that. It does seem sort of counterintuitive when you’re first introduced to the idea -

Dave: Yes.

Len: That like we should be making all these changes all the time, and we should be doing little experiments and popping things out and popping things in and seeing what happens. That seems risky and dangerous. But if you’ve got - you said sort of taking as much of the work out of it as you can, I think?

Dave: Yes.

Len: If you’ve got in built-in automated processes that make it safe to do that, like that basically will - like basically I think the term you use in your handbook on Leanpub, the “falsification -“

Dave: Yes.

Len: If you’ve got falsification mechanisms in, then you can run your, you can try these experiments safely -

Dave: Yes.

Len: Without, knowing that they won’t break anything, knock on wood.

Dave: Yes, absolutely.

Len: Yeah.

Dave: So one time we decided, at LMAX, with our system, we were running a little under 100,000 tests every hour on our system. So if we committed any change to our system, within an hour we’d get an answer, “Is it releasable or not?” With about, a bit less than 100,000, but in that kind of ballpark.

So one day we decided we didn’t like the commercial terms that we had with a relational database vendor. So we went and looked for one of the open source ones, we pulled down that relational database. We installed it in the software, we automated it in the way that we would. We put it through our deployment pipeline, which is the mechanism that runs all of our tests and tells us whether it’s releasable or not, which is what my Leanpub book is about.

We got an answer saying there were some bugs. So we fixed the bugs, we put it through the pipeline again, all the tests passed.

The next time we went into production, we went into production with that change. I’ve seen places that couldn’t do that in six months with a team of people. It took us four hours. We did it in a morning, the whole story that I just told you.

That’s the kind of change that this can make. It requires lots of changes of the way that you think about and approach software development to achieve that, but the benefits are enormous.

We were in production. You said about this confidence that you get with the validation. We were in production for thirteen months and five days, with thousands of users, before a single user noticed a problem in production. I’ve never worked on software like that before. I’m sure that people don’t believe me when I say that, but I promise it’s true. We found defects in production, but they were so subtle that nobody else was seeing them. We noticed them before other people, because of the quality of our testing, and the extensivity of our testing. So we could make changes with a huge degree of confidence, that we could make changes, and it would be safe. This system was dealing with literally billions of pounds worth of other people’s money.

Len: As you’ve mentioned, for anyone interested in sort of getting into the details of building one of these amazing pipelines, you can check out the book on Leanpub. It sort of talks about the tools and things that you can choose and things like that, but really gets into the principles behind it, so that if you want to be able to achieve this amazing thing -

But just on the subject of books, you mentioned an interesting - there’s all kinds of analogies between the writing that a programmer does, and the writing that a book author does. My favorite one is to explain to book authors what programming writing is like is, “Imagine if you had one typo in your book, and so no one could open it.”

Dave: Yes.

Len: Like that. So you get - sort of like how you work on that type of writing might be very different from how you work on book writing. But book writing also is something one takes very seriously. So you got into book writing around the sort of - you said, I think, it was when you were at Thoughtworks?

Dave: Yes.

Len: Then you wrote this book, Continuous Delivery. I’m just sort of curious about that, just shifting to the next part of the interview where we talk about your work as a writer and a content producer. Were, had you been a writer before that?

Dave: No, not really. I did the kind of writing for work that most people in my kind of profession do. Sort of writing specifications or reports, or bids for work or design documents, those sorts of things.

But I wasn’t - I’m an avid reader, but I wasn’t a writer. I’d written a few small articles for computing magazines, and, as they were then, and a few online places. This was kind of the early to mid, the early 2000s, around 2004/5-ish at this point. We’d come up with this really cool way of doing things on our projects at Thoughtworks, and I was leading one of the bigger projects at the time, which is where some of the ideas for Continuous Delivery were born.

So there was a bunch of us that were kind of mining these different ideas, and we said, “We should get together and we should all write a chapter each, or something like that.” So my co-author on that book, Jez Humble and I, sat down and started writing. We were looking around, “Where’s everybody else?” Nobody else wrote anything. We kept chasing them for a little while, and then we gave up and said, “Let’s us do it and we’ll forget the others.” So it just kind of progressed from there.

It was hard work. Writing a book is hugely hard work, but I do love it in a way. I love the way that writing focuses my mind and makes me understand things better. Because you’re trying to explain an idea to somebody else, I think you learn it more deeply, you understand it more deeply, in trying to do that.

That’s the - I don’t write fiction. I write technical books, and - but - so, I think it’s true of that kind of writing, and I do, I really enjoy the process. I just like sitting down and typing, and just trying to organize my thoughts that way.

Len: Yeah, it’s interesting you mention enjoying it. That’s one thing people often say about certain types of work is that you - in order to do it well, you really have to enjoy it sort of as an activity in itself.

Dave: Yeah.

Len: It’s sort of interesting. I mean, you mentioned you started the book in the sort of mid 2000s but it came out in 2010.

Dave: Yeah.

Len: So that was many, many years that you’re working, and with no sort of like audience, or anything like that -

Dave: Yes.

Len: Other than maybe editors, or sort of people you got to sort of read it along the way. So it is something that you really have to do, enjoy in itself. So when it came out, it became this great big bestseller. Was that a surprise?

Dave: Yes, it was a very big surprise. So I can remember sitting in a pub with Jez a couple of times, and us just talking about, “Oh, I wonder if anybody will buy it at all? Will anybody get this? Because we think it’s good, but it’s a bit nerdy, this book.”

It was an unusual book. I think it’s still an unusual book, because it’s not a, “Learn how to do this programming language,” or, “Learn how to use this framework,” or - it’s not that kind of book. It’s kind of, it’s quite wide in its territory in terms of the scope that it covers of software development. Because it’s really about, pretty much all of software development, is really the way that I think of continuous delivery. You’ve got to optimize this whole process.

So it’s an unusual book, I think. Or it certainly was then. So we were a bit nervous about it and at one point, I remember having a conversation, and Jez said, “I’ll deem it a success, if I’m able to buy a new shed from the proceeds.” So we had relatively low ambitions at that point. But it was a big success. My publishers told me last year or something, it was voted in the top 25 best software books of all time.

Len: That’s amazing.

Dave: Which was astonishing. It’s unbelievable. So I’m delighted that it’s been such a big success, and it’s had a big impact on my life and my career since, as a result of that.

Len: Speaking of delightful successes, I see a silver plaque there behind you with a little, the sort of little YouTube play button on it -

Dave: Yes.

Len: Which I gather you get, if you get above a certain number of subscribers on YouTube, or some measure of engagement, or something like that? So congratulations on that.

Dave: Yes, it’s a plaque for 100,000 subscribers.

Len: Yeah, could you -?

Dave: So that was another accident, really. So at the start of the pandemic, I was - at that time, my life was travelling all over the world, and working for companies. So my carbon footprint was horrendous, I was spending large parts of the year living in hotels and doing work.

The pandemic came, and that all stopped. My business, which was based on me doing that sort of stuff, kind of ground to a halt for a moment, so - my son was made redundant from his work, and he was in marketing. He was doing digital marketing.

So we kind of, we were bemoaning this - I think it was the week after the pandemic started, and said, and I said, “Oh, I always kind of fancied just trying to do some YouTube stuff.” He said, “Oh yeah, why don’t we give that a go?” So we started up, my wife, my son and I, started up this, we called it “Continuous Delivery Online.” We had a business called “Continuous Delivery Limited,” we still do. But we, internally, we called it “CD Online.” We started doing YouTube videos, and we released one a week. I’m very proud that two and a bit years later, over two and a half years later, we haven’t missed a week yet. We’ve launched at least one video every week, Wednesdays at 7 o’clock UK time.

We started off, and we were doing well. Most YouTube channels don’t get more than 100 subscribers. In our first year, a bit less than our first year, we were coming up to nearly 2,000 subscribers. We were all having bets between us of whether we’d hit 2,000 by the new year or not.

Then the Cyberpunk video came out, and I did a critique of the software engineering in the Cyberpunk team, and that went crazy. We got half a million views on that video, and that kind of catapulted us up to the next level.

So we sailed past our 20,000 subscribers mark, and got to 2,000. We’ve kept going from there. So we’re about, we’re just approaching - we’re a few weeks off probably 140,000 subscribers now. It just keep growing. It’s been remarkable, and it’s opened all sorts of doors, doing all sorts of interesting things.

So now we’re adding more content to the channel. So we do the weekly stuff, but we’re doing smaller things more frequently. As you mentioned, we do the Engineering Room series. So we’re trying to build out this channel about software engineering, and talking about it. We kind of sell training courses that kind of reinforce the stuff. So it’s kind of taken a lot of my work online.

I still do consultancy, but my time is kind of a bit more precious, because I’m doing a bunch of other things. So we’ve got, we are making a living from YouTube. So at my stage of life, and my point in a career, it feels rather weird to think of myself as a YouTuber. But I am these days, and that’s probably, it’s starting to overtake my book as what I’m best known for, I think?

Len: Yeah, it’s really fascinating. I was sort of like watching some of your videos, including your latest one on end to end testing, and I love the sort of self-presentation of it as like opinionated videos about things. So you set it up that way, that like, “I’m going to take a position on something.”

Dave: Yes.

Len: The thing I love about it is it’s so diplomatic, right? It’s like, it’s saying like, “This isn’t me kind of like - this is coming from my perspective and I’m going to give you an opinion on it.”

Dave: Yeah.

Len: “It’s not the same thing as I’m saying, ‘everyone else is wrong,’ or something like that, I’m - But I’m opinionated.”

Dave: Yeah.

Len: It’s this great mixture of - because they’re very - like I guess it’s interesting because you like, they’re presented as fun, but it’s - if you just read the words, it would be like a sort of sophisticated essay on something, if you know what I mean, right? Like and it’s got this great combination of those two things. So it’s kind of like, easy to sort of get into it, and then you sort of realize, “Oh this is actually a very serious topic that’s being talked about at a very high level.”

You learn a great deal from them. But at the same time there’s sort of like a fun graphic, like, “Look at this.” it’s just great, and sort of, and congratulations to your son and your wife as well for this great collaborative project. Because it, I mean, it’s really -

Dave: It’s been a huge amount of fun. But thank you very much for saying that. I’m delighted that you think that. I’ve started to think of what it is that I’m doing is rather like the popularizers of science, people like Brian Cox or David Attenborough. I’m not putting myself in the same territory as them, of course. But in trying to talk about complicated ideas, but in an easy to understand way, is what I’m trying to achieve with what I’m doing in my little niche.

I love it, and I do really enjoy it. I get such fantastic feedback from people in our social media communities now. Saying things like, “I tried this idea that you talked about, and it worked. I got these results, and this is what happened.”

So it’s just so rewarding. You get some trolling, and you get some bad stuff, it’s the internet, but hugely, massively, overwhelmingly, the feedback I get is positive. It’s just so rewarding. If I can help some people just do a bit better job of software, that’s a lovely job satisfaction for me. That makes me feel good.

So we put a lot of content out. But one of the other things that’s interesting, we were talking about writing earlier, is - I didn’t start off thinking like this, but it’s writing. So the stuff that you’re talking about is writing. So I’m writing to a deadline every week. I’m writing a new video on a new topic every week and doing a bit of research, and so on. Trying to get some forms of words that are nice and concise and clear, the ideas. Then designing graphics, and how are we going to support the picture? That’s, again - so I’m doing a lot of writing, even though I haven’t got a book project on the go at the moment.

Len: Just in the interest of time, the very last question that I save for the end of every episode, if the guest has actually written something on Leanpub, is - I checked, and I saw that you wrote, you and your wife both wrote the book in our Browser writing mode.

Dave: Yeah.

Len: But the last question I always have is if there was one magical feature that we could build for you, or if there was one terrible thing that had you shaking your fist at Leanpub every time you went on there that we could fix for you, can you think of anything that you would ask us to do?

Dave: On the whole, our experience with Leanpub was wonderful, and the next book I’m writing is going to be another Leanpub book, which we’ll be starting probably in the - towards the end of this year, or in the new year. I think Leanpub is fantastic, and for many of the reasons that you mentioned before.

When we were talking about traditional publishing and feedback loops, and being able to try ideas, and get people and share your ideas with other people, a fantastic system.

The one thing that I wish that you had that would make life easier, is a better markup validator. So that you could do more interactive editing. You could check it more - I might even - if I can find the time, I might even write one, because I think that would be a nice addition.

I know there are a few that I looked at while we were writing our book, but none of them kind of worked in our writing mode. But we found it, but we kind of tried some different things, but we did a lot of writing, as you say, in the relatively basic tools.

But I think the Leanpub experience is a wonderful one, and I really do, I really do value it. It’s been, fantastic experience with the Continuous Delivery Pipelines book.

Len: Oh, well thanks very much for that, and for the kind words, and thanks for that feedback as well. That is something that you’re probably not surprised to hear we’ve had other people have mentioned before. I mean, basically, just to say it very briefly, if you’re writing a Leanpub book, I mean, you can use Leanpub by just uploading books you write, you create using your own tools. But if you write a book using one of our writing modes, you write in plain text, basically a sort of Markdown for books that we call “Markua.” But what that means is that - currently the only way to kind of see what it’s going to look like, is to do the whole “click the button to create a preview, and build the whole thing.”

Len: So you might have tried some new little, some sort of Markua kind of thing, like oh, inserting an image. You don’t know if it’s going to work, until you click the button and create the whole book. So and - the whole ebook file. So that is something that we’ve been asked for before, and something that I’m quite confident we’ll get to one day.

Well, Dave, thank you very much for taking some time out of your evening to be on the podcast. I really enjoyed our conversation. Thanks very much for being a Leanpub author, as well as all your other work.

Dave: Oh, it’s my pleasure. Some really interesting questions, thank you for a very interesting conversation.

Len: Thanks.

And as always, thanks to all of you for listening to this episode of the Frontmatter podcast. If you like what you heard, please rate and review it wherever you found it, and if you’d like to be a Leanpub author, please visit our website at leanpub.com.