The Leanpub Blog: On Writing, Publishing, Self-Publishing and Ebooks
published Aug 06, 2016
Janelle Klein is the Austin-based author of Idea Flow: How to Measure the PAIN in Software Development. She is the CTO of the recruitment consultancy New Iron and founder of Open Mastery, a network for peer mentorship based on data-driven mastery of software development.
In this interview, Leanpub co-founder Len Epp talks with Janelle about her career, the ideas and experiences that are the inspiration for “Idea Flow”, the concept of “Idea Flow” itself and how it can transform the relationship between developers and management to profoundly improve productivity and quality, and at the very end, Janelle also talks about her experience self-publishing on Leanpub.
This interview was recorded on May 27, 2016.
This interview has been edited for conciseness and clarity.
Len: Hi, I’m Len Epp from Leanpub, and in this Leanpub Podcast, I’ll be interviewing Janelle Klein. Janelle is an Austin, Texas-based NFJS tour speaker and technical mentor. She’s the founder of the Software Mastery Circle [which is now branded as “Open Mastery” - eds.], which is dedicated to data driven software mastery and aligning the interests of business and software engineering. Janelle is the CTO of the recruitment consultancy, New Iron, that specializes in building software teams.
Janelle is also the author of the Leanpub book, Idea Flow: How to Measure the PAIN in Software Development. The book is about the hugely important topic of technical risk management and presents, in Janelle’s words, “A modern strategy for systemically optimizing software productivity with a data driven feedback loop by measuring the pain, or friction, in developer experience. People can identify the biggest problems, understand the causes, and run experiments to systematically learn what works.”
In this interview, we’re going to talk about Janelle’s professional interests, her book, her experiences using Leanpub at the end of the interview - and ways we can improve Leanpub for her and other authors. So thank you, Janelle, for being on a Leanpub Podcast.
Janelle: Thank you for having me.
Len: I usually like to start these interviews by asking people for their origin story, and I was wondering if you could tell us a little bit about how you first became interested in Agile and software development.
Janelle: Sure. I’ll start from college - well, slightly before college. When I graduated high school, I ran off and married my high school sweetheart. Not the best decision I ever made. But that got me running off to California to be a professional song writer and following my dreams.
And then I got into college, and I started to realize what a career in music might actually be like. And it just sort of sucked all the passion out of me, thinking about having to write to make money. And I knew absolutely nothing about software development. We had a computer growing up that I played lots of games on. So I’ve always been a huge, avid gamer.
And my ex-husband at the time - he was in the military doing network management, computer-y stuff, kind of a hardware geek - he’s like, “Let’s take this assembly programming class. That’ll be so cool. Assembly’s like awesome. It’s all this low level cool shit, right?” And so I was like, “Okay. That sounds like fun.” So we went and took this assembly programming class, and I was like, “Programming my calculator in math class, I can figure this out.”
I got basic instructions, and I started writing reams and reams of assembly code. I wrote the game, Breakout, with the paddles, and all that, completely in Assembly, with the full 256 colors and little music and beeps when the thing hit the wall - just completely in Assembly. And my teacher was like, “Well, at this point, just show me what you’re working on. You can work on whatever you want, and you get an A.” And I’m like, “That’s cool. I should take more classes like this.”
But that moment really changed things for me, because it was this moment of unlimited exploration - where I can go and create anything I could dream of. It was this ultimate kind of artistic medium, and that’s when I fell in love with software development. I’ve had one experience after another, that’s been this giant open-ended problem of following whatever dreams that you might have, and turning ideas into awesome tools. As I’ve learned how to do that with other people, it’s been one of the coolest experiences imaginable. And really, software development is very much a love of my life.
Len: That’s a fantastic story. When I’m interviewing Leanpub authors, many of them are in software development, and it’s curious - I think at this point, it’s now less than half of the people I’ve interviewed who end up in software, took something like computer science in university. People come into software development from so many different directions. I was wondering how you ended up at New Iron doing consulting work?
Janelle: So I ended at this kind of strange company, New Iron, doing - I mentioned it’s a software niche recruiting company, or you mentioned that in my intro - I started working for New Iron just as a contractor on this project at a semiconductor company. I got really involved in semiconductor, and lean manufacturing and supply team optimization and process control. And doing back end, high volume data automation stuff, and problem-space wise, it was really cool.
But as I started to learn more about lean - because the company was all about everything lean - I started to get involved in their lean consulting program, and teaching lean practices around the company, but in the context of software development. After figuring out how to solve the problems on my own project, and just getting better at software development, and how to figure out what your problems are, and turn around a failing ship, I got really good at helping other people to solve those problems.
And so it was never really that complicated in consulting. The main thing I discovered is that the job of a consultant is essentially to come in and identify the elephants and point to them. Usually, if you just listen to what people are saying, they’re already talking about what all their problems are. A lot of the time, I’ll just echo the same things that the team is already saying, and just explain it to management - but with a nice Keynote and explaining things in management-speak.
And suddenly all these communication problems where engineers couldn’t talk to management became my niche. It was figuring out how to communicate all this pain with metaphors and stuff, so that I can bridge that gap. A lot of the failures of projects are caused by these problems, with this inability to communicate. And so, with consulting, it was kind of a natural niche for helping out with that.
And then with New Iron, since we specialize in technical assessment and developer mentorship, I’ve got to focus on teaching people, doing a bunch of stuff with community, mastering the art of technical assessment, and teaching people how to build more effective teams. I mean, it all kind of went together with consulting.
In school I almost got my PhD, but I decided not to, because I was really eager to get into industry, and I really enjoyed like all the HCI kind of stuff. I always had this love of science and research. And I thought about going back to get my PhD, and then realized, “You know, I don’t need a need a degree to be a scientist, right? If I want to go and do research and learn about problems, I can just set up my own little research lab and learn about problems.”
And so I started treating developers kind of like guinea pigs, and set up a little lab in my work environment, and then started codifying the things I was learning about how to teach the art of software development into patterns and principles using these tools. A lot of that learning created the basis for Idea Flow. So I didn’t scare everybody away. I decided to write my book as kind of a first person narrative story of my own experiences, so it didn’t read like a dissertation.
Len: I’m really looking forward to actually talking to you about your book. It’s really interesting. I just have one question I’d like to ask you before we get to that though, which is about - there’s a line, I think, on New Iron’s website that says, “Being a good developer doesn’t make you a good interviewer.” This is something I’ve been reading more and more about in the tech press lately. It’s become a little bit of a meme, I guess, especially in relationship to people with various kinds of disability. I was wondering how you would recommend people hiring software developers to approach giving interviews?
Janelle: I think the main piece of advice I would give developers, is to focus on testing decision-making abilities - rather than current, active skills. The profession of software development is all about being a professional learner. It’s knowing how to find and look up information - how to break down problems, and how to learn your way out of a broken situation.
And so what I eventually came to is, that we need a method of technical assessment that focuses on assessing a person’s capability to reason about problems and navigate trade-off decisions. As opposed to, “Do you know what TDD is?” or “Can you write a unit test?” Because at the end of the day, it’s not about what you can do right now. It’s about what you can learn when you’re on the job. Because you’re never going to be faced with the same problem and the same challenges.
I mean, this industry changes so fast. We’re professional learners. We’re professional problem solvers. And that’s what makes people good, is their craftsmanship skill, that comes from years of intuition based around problem-solving, is really what skill is. And that’s what we need to be assessing for. So the main thing I do is put real problems in front of people. And then I ask them lots of questions to poke into their reasoning process, as opposed to, “Can you get from A to B?” It’s about, “What kind of options are you exploring?”
Len: Do you recommend doing things like asking riddles to see how people respond to unfamiliar questions?
Janelle: I’m not good with riddles, so it’s hard for me. Yeah, probably not answering riddles. I would hate to be given a riddle in a - I guess the other thing I’d say, is that when people get nervous, their ability to make intuitive leaps to solutions generally shuts down a bit. But their recognition skills don’t. If you put something in front of somebody, and they don’t recognize that there’s a code smell of sorts, or some problem with things, then that’s usually a sign that something is actually wrong or missing.
Whereas, just because somebody can’t come up with a right answer - it’s just the world is not that black and white. And I know technical assessment is really hard. For example, with respect to certifications, we’ve been working on a way to systematize technical assessment, so then I could teach other people how to do my interview technique.
It’s taken us a decade to figure out how to do that. But a lot of people punt on the whole certification problem, because, “It can’t be done, it’s too hard.” There’s so many problems with commercialized certification in our industry, that everybody is like, “Oh well, we shouldn’t even try, because it’s just going to cause so many problems.”
I think we need to move toward more of a model, where it’s open certification. Where we’re testing decision-making skills, and it’s an industry public debate about working on getting better at this, as opposed to just giving up on the problem, because it can’t be done. And just make technical assessment a free standard, almost, that we work on together as a community. Because we all benefit from having a clear definition of what good is. I mean, we need that.
Len: Yeah, definitely. In that context, I’m curious what you think about nervousness in interviews and things like that? Because the interview is a situation that the person will never find themselves in again - more or less, if you hire them. They’re going to then be working. And so people can be nervous in interviews, but not nervous when they’re not doing their work.
Len: I was once interviewing someone who’d just finished a masters in maths at Cambridge, and failed to do basic arithmetic in response to a question. So I knew what the interviewee was capable in normal circumstances, but not in the stressful circumstance of an interview. And so even the best certification or qualification - or something like that - that someone comes into the room with on paper, can sometimes not translate in an interview, but can in work. I was wondering if you advocate giving people problems to take home and solve or something like that as part of the recruitment process?
Janelle: I don’t really do take-home problem solving kinds of stuff. I mean, I certainly see the benefits. But the main problem I have the take-home thing is that it takes too long, it’s too much work. In a market where everybody’s hiring, and there aren’t enough good people, the last thing I want to do is go an create more barriers to entry to hiring the right people.
And so we’ve basically focused on, how can you get the maximum amount of information to make a quality decision in a limited amount of time? Because the take-home thing has to be scheduled around life, it’s much easier to get people to show up for a scheduled one hour call. And then I basically do that with video sharing and stuff, so I can throw code up on the screen and ask people about it. We’re all about optimizing the time spent in that.
And then the other things I do, for the onsite, are a code pairing interview, and then something that’s more design-focused. So I tend to focus on core skills, as opposed to, “Do you know whatever trivia?” Like whether you know ordering of some… I don’t know? There are some things that are just - Computer Science-related kinds of questions, that are so far removed from what we actually use on a day-to-day job - I’m much more interested in people’s everyday problem solving skills. Can you refactor this code? Can you work with the people on your team? Are you an asshole? Those kind of things.
Len: That’s a really great and very practical answer. I don’t know how much of your work actually is recruiting people in Austin, but I can imagine the competition for good developers there must be really high.
Janelle: Yeah. We pretty much only work in Austin. We have some stuff going on in other states from clients that have multiple sites in most locations, but we’ve always been very central here. And then since our company is run by software folks - it’s kind of interesting how we ended up in software development.
New Iron used to be a product company that built a first generation web services stack. We’re out in California, and I thought, “Oh, we’re going to build this awesome product, and everyone will buy it because it’s so awesome.” Obviously that wasn’t a very good business strategy. And so we’re sitting around going broke, basically, and we got a call from an old client that we used to work with, and they’re like, “Hey, can you guys help us on this consulting project on the software?” And we’re like, “Money, no money.” And so, easy choice, right?
And so as we started consulting there, they’re like, “Do you guys know anybody else that can code?” And since we already had a vendor agreement set up, we just started talking and recruiting all our friends. And we did full-on interviews for everybody, not having any idea how the recruiting industry was supposed to work - that nobody does this kind of thing.
And so we started getting really good at technical assessment. And that sort of became our niche. Because it’s hiring people, and figuring out what skills you need, versus what’s available on the market - and having unrealistic expectations about hiring Superman, and that you can instantly take all these things that you’re deeply familiar with and say, “Oh, now you do this job.” There’s so many companies that have so many unrealistic expectations about it. Because hiring isn’t a first priority in a lot of companies, so people don’t necessarily put the time into it that it needs and deserves.
Yet it’s so important and causes so many problems when you hire the wrong people for the wrong things. And so we got a lot more involved in understanding what our clients actually are trying to accomplish as company, as a team, and help them to come up with a realistic strategy that they can actually find a gap they can fill, as well as the gap that they can sell. Beause in this kind of market, if you have a job for, “Oh, this is a Java development position like so many other Java development positions.”
Len: One thing one reads about in the press around recruitment is the way that employers or recruiters will sometimes look into a person’s internet history. And I was curious if that’s something that you recommend employers do?
Janelle: Internet history in terms of, like, what they post online?
Janelle: It’s not something we do in that, generally speaking, we focus on whether the developer has the skills, is able to demonstrate the skills, as opposed to any kind of past history kind of thing. I think since we have a really good process for technical assessment itself, it’s like everything else is almost irrelevant. In that, if you’ve got one year of experience, but you can bring it when it comes down to sitting in front of a computer and solving a problem, you know what? I don’t really care if you’ve only been programming for a year. I’ll just think you’re awesome.
Len: That’s a really fantastic answer, I’m glad to hear that, that sounds very reasonable.
Moving on to the subject of your book - which, by the way, I really liked - you start with a pretty dramatic story about your experience working on a statistical process control system for a semiconductor factory, I think. And you mentioned the elephant in the room before. In that particular experience there was, I guess, a sort of invisible elephant there. I was wondering if you could talk a little bit about that experience and its impact on your approach to complex software engineering projects?
Janelle: Wow. Okay, that’s a big question.
So in terms of the experience, I was on this project with this really great team - really smart people. We had test automation, CI, great team, disciplined practices. And despite doing all the things that we’re supposed to do, our project basically crashed and burned and hit a wall. And by hit a wall, I mean we brought down production three times in a row, and then didn’t ship again for another year.
And the third time this happened - this is a 24/7 operation, so downtime is a really big deal. And the last release, the third time we brought down production. Not only did production go down, the rollback failed. And then the feature toggle that disabled the broken stuff failed. And so we were up at like three in the morning hacking out a patch fix to, just so we could get the software out of production while everything was down. It was the most miserable experience you can imagine.
And this was shortly after I’d been out of college. This was my second job out of college. I was kind of a little hotshot, and had my head in the clouds, and thought I was awesome and that I was pretty much infallible. And I had this great idea that would improve performance to basically flip the architecture inside out and do this massive set of changes. And, of course, I put that all in production all at once.
It’s just one of those huge lessons in your career that you never forget. And that really fundamentally changed the trajectory of my life. Especially since all these things, I was like a super TDD zealot-type at the time. And so I was convinced that our success was all about whether we were following the right practices, and whether we were doing disciplined TDD in our project. That was the thing that mattered.
And when I was doing all this great stuff, and I failed anyway. It was the most foundation-shattering experience that I could’ve ever imagined. And so we got together as a team, and we’re like, “What the hell are we going to do?” Because we’ve been doing everything right, but we’re failing anyway. And so we built this tool that could detect high risk changes in the code, because we thought the problem was technical debt building up over time. Same problems we always talk about are causing the pain.
We thought we could use this tool to let us know where we needed to do extra testing, by identifying the areas where there are changes by tech debt code. But what we found was that most of the mistakes were actually in the most well-written parts of the system. And so we started digging around in the data, trying to make sense of what we found. But there weren’t any answers that made sense and correlated with what we knew about software development, and what we’re supposed to be optimizing for.
The correlation we did find in the data was that a lower familiarity with the code tended to increase the likelihood of mistakes. And while that made some sense, it seemed like there had more to the story than that. Because when we had to work with complex code, it was really painful. And so we started down this path of trying to understand, “Well, what it is that makes development feel painful?” And so as we are working, we started collecting all this data on where we were experiencing pain, and then talking about the specific things that were causing the pain, and keeping measurements about how long all these various things took, and just writing up the things that seemed like our biggest problems on a whiteboard.
It wasn’t until we started down this road of collecting data, that we realized that we’d been essentially solving all the wrong problems. We had all this test automation, but the test didn’t catch our bugs. And we had well modularized code, but it was still really difficult to troubleshoot all the defects. And then we started looking at the specific things that were causing our pain, and learning with the data-driven feedback loop, we realized that most of the problems were caused by human factors that were, how a human interacts with the code while they write.
And so that’s where this whole technique of measuring development experience came from as a means of feedback. Because it was the only thing that seemed to really matter at the end of the day was whether it was easier or harder to do our jobs. And if we measured that, data became this unifying force on our team where it really brought us together in learning how to learn together.
And at the time I had this experience, I couldn’t put what I’d learned into words. But I knew that I wanted to teach it to other people. And so after I read Peter Senge’s book, The Fifth Discipline, about how to build a learning organization, I was so inspired by these ideas that I was like, “Alright, I need to figure out how to write this down.” Because this is the essence of what Peter Senge is talking about in his book – these five disciplines of a learning organization, and learning how to learn together.
Essentially, I found this really tight correlation between science and learning organizations. And then when we start using this scientific method-ish process to learn and examine the data and have a shared direction of better, that we’re all reaching towards, that suddenly it unifies a team in a really cool way. And I’ve been fortunate to have some really incredible experiences working with just amazing engineers, where we learned how to learn together as a team.
And so it wasn’t about what practices we were using. It was about whether we could identify the right problems to solve. And once I learned how to identify the problems, the rest of it was easy. I got a team of professional problem solvers. So that turned into the skill that I wanted to figure out how to teach. Even though it was hard - even though we didn’t have words in our vocabulary to describe the things we’d learned - I was determined to figure out how to do that, and to teach these things to other people.
Len: And when you talk about the data gathering, I know from reading your book that there’s very specific things around that. I was wondering if you could explain a little bit about what the data is, and how the gathering takes place.
Janelle: Sure. So I’m measuring, specifically, troubleshooting, learning, and rework. And we have tools that integrate with the developer’s IDE that are partially automated, partially manual. From a time-tracking perspective, it’s a lot different than managers wanting us to track our time in JIRA, because there are tools created by developers for developers to use to understand what their problems are.
So we’ve always kept management out of the data. It’s like, “We’re doing this for us, go away.” We’ve come up with better ways to automate things over time, but essentially - let’s say, when we’re writing code, we go through this cycle of, you write a little code and work out the kinks over and over again. And when you get to that point after you write a little code and you’re validating things, you’ve kind of got a prediction in mind of what you’re expecting the code to do. And at that point, there’s a decision point in our brain. If the code does what we expect it to do, we write a little bit more code. And if the code doesn’t do what we expect, we go into this troubleshooting mode.
And what I found is that, moving forward, modifying more code, learning - basically figuring out what you’re going to do, can take a whole lot of time, especially now that 90% of our software’s built from existing parts. Or, if you’re unfamiliar with the code, learning is a substantial part of our work that gets in our way. And on the troubleshooting side, we’ve got to troubleshoot the problem and then do rework to put things back in place before we can move forward again.
And so this flow of ideas between the developer and the software, and the friction in that flow - I’m measuring in terms of troubleshooting, learning, and rework, and then actual duration of those times. Then, we have a general rule on the team where anything that takes longer than 20 minutes, we talk about, and try and understand what the causes are that made this incident so painful. And it’s always a kind of gut-feel type factor. In some one case, 20 minutes is not much time. In another case it’s way more time than it should be. And so you kind of just talk about the things that are worth talking about. Don’t talk about - it’s just kind of noise.
Len: I find the importance that you place on pain to be really fascinating. I have a sort of specific question I’d like to ask about that. But first I’d like know if you could maybe define a little bit about what you mean when you talk about pain?
Janelle: So in this context, I’m relating pain to friction in idea flow. The reason why I use pain specifically is for a couple of reasons. One is that I want people to focus on the symptoms, as opposed to the cause. Because one of the problems we having our industry is automatically assuming what the cause is of the pain - like it’s this ugly code I have here, as opposed to focusing on the symptoms, and our experience, in working backwards.
And so, I use the word pain specifically to get people thinking about, “Well, what are the symptoms?” And then let’s have a richer conversation about what are the potential causes. Because usually most problems have multiple factors. And I think that’s one of the places that developers tend to get stuck on best practices, as it’s very solution-focused.
The other thing is that intuitively - just from experience - pain is associated with this idea of something that we want to avoid, and figure out how to have less of or reduce it. And from an optimizing, idea flow standpoint - that’s kind of the message I want to relay. There’s this thing that is a threshold thing that we want to keep in check to a level that we can deal with it. It’s not about eliminating pain. It’s about figuring out how to optimize and manage the pain so it’s at a tolerable level, and stays that way.
Friction - even though I’m associating it with pain - it often doesn’t totally correlate with what developers are using that word for now. For example, one of the perceptions that’s usually really far off in software development, is just our notion of what takes a long time. And one of the things that happens is when we’re busy working on a problem and doing stuff, time seems to zoom by really quickly. Whereas when we’re waiting on something to happen, like waiting for something to execute, it feels really painful, because we have to wait all that time for that thing to get done. And time seems to go really slowly.
And so when you start looking at things in terms of elapsed time, we feel like that thing that took five seconds was like an eternity, and that that was really painful. But then we’ve got all this time that we spent troubleshooting, that we don’t necessarily associate with pain - even though it takes an order of magnitude more time, in terms of human cycles. But because we’re busy doing stuff, and problem solving is fun even - doing things that are fun, we seldom associate them with being painful, right? And so I was trying to create this mapping visually, to sort of remap our pain sensors to something that is explicitly defined, that we could all share this universal definition of, What is pain? Being things that slow down the process of idea flow, even though it takes some remapping of our pain sensors.
Len: It’s really interesting to me. One of the reasons why I liked your book so much, was that you talk about addressing pain as a problem, and you were saying just now that pain is something we want to avoid. One of the reasons I find it so compelling is that, that’s so often not true in management practices.
My brother told me a great story where I had a kind of apple-falling-on-Newton’s-head moment. When he walked into a big box store, there were two employees - one of an earlier generation, and one of a later generation. They were handing out flyers or something, wearing little vests. The younger person started walking off, and the older person said, “Where are you going?” And the younger person said, “To the bathroom.” And the older person said, “Make it snappy!” What occurred to me in hearing that story was that pain is something that people often use in managing others - in a sense, positively, right?
Len: They use pain as a tool to get the work done. I was thinking about what I’ve read about slavery, and the way there’d be an overseer watching people work, and cracking the whip, and making sure that they stayed active. And I just saw this line from that all the way to this box store, and these two people in vests, and in the way observation, and the infliction of pain is so baked in to various kinds of management practices. We don’t even kind of notice it, it’s so obvious.
I was interviewing someone recently, who said, in his first software job, he was a tester for the Australian government. [Editor’s note: this was actually at a call center, not as a tester for the Australian government. Sorry, Australia!] And whenever he left his station, he had to log why, including going to the bathroom and for how long - the humiliation built in to that.
Anyway, just to finish, the thing I find so interesting about managing software - and especially in a world where software is eating the world, and everything is being driven by software, it’s become this really important form of labor, but it’s invisible.
For example, imagine you’re in that long ancient tradition of the manager who watches, right? It doesn’t matter if you’re watching people sowing seeds or reaping the wheat, or working on a factory floor in an assembly line. You can develop all these practices around watching people. But with programmers, they’re just sitting there, right?
You can imagine being from that tradition, and then you get hired to manage programmers and you’re like, “Well, what’s there for me to do?”
What you’ve done in your book and your ideas is that you’ve like completely flipped it around, and abandoned this form of management - that’s so universal and so inappropriate for a world where software is important. People have attempted in so many ways to try and do metrics and things like that around the wrong things. And what you do, in my opinion anyway is, that you’re looking at exactly the right the thing, which is taking into account like the pain that people feel when they’re doing types of work, and then really thinking that through. So, anyway, I just wanted to say, I think that what you’re doing is actually quite revolutionary.
Janelle: That’s an interesting story and take on the management side of this. One of the other big target audiences I had in mind when writing my book was managers, and helping them to get a feel for what it’s really like to be a developer in this world. I don’t think - if you listen to the way managers talk and the way developers talk, it’s almost like we’re speaking two different languages. Developers use words like “beauty” to describe the code and “elegance” and “firefighting” and “explosions”. And all of our metaphors are in terms of art and war. And if you listen, it’s all about money and profit and bonuses and ROI, and all this money-related stuff.
And I think what we have to realize is that as software organizations, we’re basically building a business that sells profitable artwork. You need to think about the economy as a giant art contest. With software development, it’s very personal in that we very much invest our hearts and souls as software developers into our work. And then seeing your creation get stomped on by all this organizational dysfunction is an emotionally damaging experience.
And so there’s all this cynicism and struggle, and a feeling of helplessness that turns into contempt within our organizations. And we get these divided cultures, where you’ve got a management culture that’s all trying to raise morale through celebration. And there’s all this fake happiness, for a lack of a better word. And then defensiveness around not wanting to shatter the perfect illusion bubble. And so management is often really resistant to talk about problems and pain and things that are going on. Because then it seems like an reflection on them if they’ve got little red marks on their status updates, whereas–
Len: I can’t stop thinking right now, when you’re talking like this, there’s this infamous photo of the Yahoo executive team dressed up in Wizard of Oz outfits, that was being presented to the employees as this sort of like motivational thing. Like the leaders are all on board and sort of superstars, and then the total disconnect in the use of that image, and what’s really going on there.
Janelle: Yeah. I see so many managers that are really concerned about talking about the problems, and that they believe that it will bring down morale if they talk about the problems. But in fact it’s quite the opposite that usually occurs. Especially when you have data to have a conversation about, putting the pain on center stage, as opposed to fighting each other, and blaming other people for the problems - you have a beast to conquer. And then when you go and conquer the beast, then it builds camaraderie within your company, and it brings people together around seeking truth, as opposed to this culture of imagining the world that we wish we had and trying to will that into existence through celebration. Or the after work drinks and bitching about our jobs, and how nothing’ll ever change and how everything’s hopeless, which starts to breed this undercurrent of contempt in our organizations and, emotionally-speaking, people are very much going to war.
I see a lot of these problems in organizations across the industry, where our software problems create cultural problems, and have a huge effect on the lives of human beings. I’m hoping that these ideas catch on from a standpoint of a strategy where we can bring the art side of the world together with the money side of the world, and realize that we can have both.
Granted, it takes a lot of work to figure out how to get there and how to learn together to optimize the whole. But at the end of the day, you build something beautiful. And software organizations become about working together to bring something awesome into the world. And then when you’re competing in the world’s art contest, you make money because you built something awesome. At the end of the day, I think those are the kind of things that human beings really care about - it’s having the opportunity to do something that matters, and to experience that with other people.
Len: It’s sort of central to your vision, your insight, into what people are actually doing when they’re engaged in software development. It flows directly from your basic premise - that things like pain, and things like the human element matter because it’s a human activity. It’s so interesting that we live in a professional world, where talking about the human element immediately requires a little bit of defensiveness or something like that, right? Because that’s supposed to be of secondary importance, and just a problem to avoid.
But your idea flow metaphor puts the human activity - it’s sort of like ontologically at the heart of what is really happening, because, as I understand it, the idea is that - well, I’ll quote you here: “That interaction with the code as a stream of ideas, flowing between me and the software, is a correct description of what’s actually happening in coding. That I have an idea that I try to express it code. And then that code is then used to carry out various things. And you can see whether my idea was translated properly by me into the code, and then properly read by the machine reading the code. And then the next step is when someone else is reading my code, have I correctly flowed my idea from its origin within me into them?”
I just find that so compelling because that actually is what’s happening, that we need take into account. I mean, one can say the human side of things, but just that it’s consciousness that is engaging with the machine through code. And to try take and consciousness out of it, is to completely misunderstand what this process really is. And that’s why it flows naturally that concepts like pain actually become scientific metrics to measure, because that is actually what you’re looking at - is a process of consciousness engaging with something.
Janelle: Yeah. One of the interesting extensions - I mentioned early on, using mentorship, and creating a developer lab to test these ideas - one of the ways I’ve been looking at idea flow as mentorship being, I have an idea in my head about how to optimize productivity, say. And how do I teach that idea to somebody else? That’s another type of idea flow, of sharing that idea. And the thing that I learned from that is that the only real difference between communicating something and teaching something is whether we take responsibility for whether that other person actually understood the idea or not.
And so, likewise, I came up with a method of testing whether a person understood by testing their ability to think through and make trade-off decisions in a variety of staged scenarios. That led to my technical assessment technique, too. This whole underlying concept of “What does it mean for an idea to get from my head into somebody else’s head, and how do we come up with an explicit, sort of scientific, method for testing whether that has actually occurred or not?” - has given me a feedback loop for learning how to become a better mentor as well.
Len: On that note, you talk about communication, which becomes very precisely defined in this kind of circumstance. You talk about the importance of metaphors. I was wondering if you could talk a little bit about why you think metaphors, and good metaphors, are so important for this process.
Janelle: So, I had a couple key influences with that. One of them is a book I mentioned in Idea Flow called Metaphors We Live By, which is a theory on human understanding as a fabric of metaphor, and that essentially everything we understand is in terms of other things we already understand, and that we build up understanding through relationships of metaphors. And so one metaphor, how it relates to another thing through metaphor, and then how those things are similar and different.
The book goes through a whole bunch of a examples in our languages, and comparing cultures, and how when you have a different foundational metaphor, you get a fundamentally different understanding at the cultural level. All of this cultural difference analysis - it’s a fascinating book, and it fundamentally changed the way I started looking at learning and improvement and communication, in that I started to think of my brain like a shape toy, and that I’ve got all these shapes that I know in the world. When I see a square, I sort of shove that into my shape toy brain, if you can imagine that. And in order to be able to recognize something else from my experience, I basically need to define a new shape. And so one of the feedback loops I discovered was that by explicitly defining new vocabulary - like looking at my experiences and then saying, “Oh, this is this pattern” and making up all these new pattern names for types of mistakes or for different strategies or problems in the code or that make troubleshooting take a long time or whatever it is - I basically built a massive taxonomy of patterns.
But in the process of doing this, I realized that I started to recognize things in my experience that I never noticed before, and that there was this feedback loop of, the more patterns I could develop in my vocabulary, the more detailed my observations were from my experience, and the faster I ended up learning. It was like I had amplified what I could learn from each experience, as well as giving myself all these awesome tools and connections for synthesizing information I’d learned.
And there was one metaphor that had a huge impact on my thinking. So Ash Maurya - he’s got a book coming out really soon called Scaling Lean - I went to a workshop of his like three years ago, when he first started working on this book, and he introduced this metaphor he calls “the customer factory”. So Ash focuses on lean startup practitioner-related material.
Essentially what he’d done is codify a method for practicing everyday science, and then mapped lean manufacturing to kind of a customer experience model. So if you imagine your business is like an amusement park, and you’ve got people lining up to the entrance of the amusement park, when they want to use your product and then, their first experience of taking the rides - that kind of thing. And then he came up with this way to measure and model the flow of different experiences through the business, and measure that and optimize what he calls “happy customer flow”.
And what was so fascinating about this, is it was like an experience business - sort of an experience-based supply chain model. And since all the stuff I started doing was in developer experience, as I started looking at the problems, I figured out how to build a supply chain model, mapping the same kind of metaphor idea to software development. But rather than the process flow that we normally think of as the supply chain, I was using the software dependency supply chain with experience-based data on top of it - how you experienced changing a particular software component, for example, and modelling across software dependencies.
And once I figured out how to do that, and optimize idea flow across the idea flow factory as it were - since my background is in process control and supply chain optimization from the manufacturing world - suddenly it was like everything I had learned from lean manufacturing, like clicked together with everything from Lean Startup. That clicked together with everything I’d been doing with idea flow, snd it was like this massive synergy of patterns that suddenly went together, and they were all linked together through metaphor.
Whenever I discover a new metaphorical link, it’s like all of these new discoveries and ways that you can make associations just fundamentally shifts. And I think the major revolutions in our industry always come with some kind of metaphorical shift.
Len: That’s a really fascinating way of putting it. I mean, you used some great metaphors in there as well - like the brain as shape toy interacting with the world. And customer experience as being like an amusement park, and that’s just amazing. I think you sort of proved your point by using those metaphors so well. Because it’s sort of like, when you successfully use a metaphor, it’s like a shortcut to communicating not just a set of facts, but a conceptual structure around viewing facts - and understanding what you’re actually looking at in a practical way.
In your book, you also talk about addressing the root cause of project failure in the software industry, and this is something I think is one of the most important things facing industry, I guess, generally today - is that as everything becomes software-driven, understanding the issues in software development becomes more important. In particular, we all see in the popular press these catastrophic software failures.
One of the ones that first come to mind for most people would probably be - or at least in the West - would be the Obamacare roll-out and how that failed. Up here in Canada, we’ve had our share of these catastrophic failures as well. I was wondering, in the context of your concept of idea flow and your book, if you could maybe explain what you think the root cause of project failure is in the software industry, and then by extension all of industry because it’s driven by software?
Janelle: I think there’s a few different core problems, but I’d say the one that’s probably more of the root would be our inability to share knowledge as an industry. We’re sharing this solution-centric knowledge with one another, and what I found is it’s not enough. The things that we really need to know in our industry are largely tacit knowledge, and they’re largely learned through mentorship.
As the industry is growing, and more people are getting into the industry, they’re not getting the kind of education they need in school to know how to do these things. We end up with more and more projects with people that don’t have the skills that they need to do their jobs, and don’t know how to communicate these things to management. And management doesn’t know how to understand what’s happening on the software project to make it be successful.
And the combination of what I’m referring to is the “wall of ignorance” between the management world and the engineering world - and how we’re basically trying to explain and describe our problems being completely understood, being one major factor. The other one is not being able to share our knowledge as an industry, or have any kind of universal definition of “better” that we’re aiming for. So it’s like we talk about brevity and modularity and test coverage, but all of these things are really kind of a means to an end, and we don’t have a way to describe what better really means, as the center point in that conversation.
I think that’s one of the big things that idea flow brings to the table. It’s that it gives us a way to share our knowledge on how to do this stuff. Because these projects that fail - it’s just sad. Especially when the taxpayers get stuck with the bill, of just mismanaged projects from people that are running things that don’t know what they’re doing, when that knowledge of how to run a successful software project, is very much knowledge that is within our industry.
If we could do a better job at, one, sharing our knowledge, and then even taking responsibility for public service based projects, and potentially running these things as open managed projects by the industry - as opposed to having the government having no idea how to build software and then running these broken software projects. It’s a tax burden and stuff that is completely unnecessary, because we can solve these problems. It’s not - I should say it’s not that software isn’t difficult, it’s that sometimes the failures are so unnecessary.
Len: It certainly looks that way from the consumer end of things.
Janelle: It depresses me a little bit that we end up wasting so much of our money on things that - for example, if we ran healthcare.gov as an open source project, I think things would’ve turned out very differently. If you basically had one, we could’ve had the thing happen for free, basically based on voluntary support. And because it was something that’s important to the people - I’ll bet you, people do a good job at managing that project. Just because a lot of people care. And we can build societal support based on just humans caring about each other, and helping each other to build the stuff that we need.
Len: It seems to me that - well, with government in particular, there’s the challenge that being skilled at getting government contracts, and being skilled at building software, are two very different, and perhaps, incompatible things.
You talk about the “wall of ignorance” between sort of people doing software development and managers. I think that from conventional management perspectives, there are so many things about software development that are just new and counter-intuitive.
For example, if you’re a typical manager - and I’m going to just stereotype negatively here - you want to have more people underneath you. “I’ve got 10 people underneath me.” “Well, I’ve got 100, you loser,” would be the talk over the bar, right? But in software, it could be that having more people is worse, and actually lowers productivity, and lowers the quality of the work that you’re doing. And you can’t just have more people doing the planting or working the machines or something like that.
For example, a person just sitting there thinking, might be way more productive than a person typing or doing something active. In a very straightforward way, less is more, right? You want the smallest code base you can get, not the biggest. A manager might brag, “Well, my project has eight million lines of code.” And you’re like, “Well mine only has 10,000.” But it could be that what you’ve achieved with those 10,000 lines of code is superior to what the other group achieved with 8 million lines of code.
And so I think that a lot of the ways people measure success are out of alignment with the ways we ought to be measuring success, and that that “wall of ignorance” is partly there. You say ROI and money are the terms that managers are living with. And then the developers are living with - say beauty and efficiency, or something like that, and it just seems to me that one of the reasons those big, especially government, projects fail, is that there’s this concept that management and work are these fundamentally different things, and that’s there from the start. So the sort of the whole thing needs to be thrown out, in order for it to be improved.
Janelle: Yeah. I–
Len: You don’t need to agree with anything I said, by the way.
Janelle: I was just thinking in terms of management. I haven’t really run into any managers that were concerned about trying to measure lines of code as a measure of progress. I think we’ve moved past that era. What I see happening more is, the first problem you brought up definitely, of not realizing the impact of adding people to the project, or assuming that you can replace one developer with another developer. And not realizing the impact and the learning curve, in the effects of familiarity on a team.
Part of the problem I think is that - back to the metaphor of technical debt, that has us kind of thinking about the problems in terms of an interest rate, or like a predictable stream of interest payments over time. And when we think about interest, we think about, “Oh, we’re paying 10% interest, 20% interest”, and it’s this steady, predictable thing. It creates the illusion that we can throw more people at the problem, and distribute the cost across more people - when in actuality what’s going isn’t a problem with a predictable increase in cost, it’s a problem with chaos and a loss of control. And that metaphor is failing to explain the true nature of the problem that we’re trying to solve.
I realized this when I was in a business coaching session with Keith Cunningham, and I was talking about technical debt. And none of these people in the class knew anything about software. But I was trying to explain it in terms of these metaphors, and how technical debt was such a bad thing and what it was like. And the response was, “Well, that doesn’t seem so bad.” And I’m like, “What? Don’t you even want to know like the interest rate or something?”
And what I learned in that world, was that, the way these guys do math, and make decisions at the investment level, just seems so crazy, because it’s so far removed. But it’s like, “Okay, we’ve got revenue minus cost equals profit, and our goal is to raise profit by 10%. Well, how are we going to do that?” Well, you have to basically come up with an investment strategy of how you’re going invest money in all these different buckets to make this prediction come true.
And what I realized in this class was that what makes investment decisions more difficult isn’t an increase in cost, it’s a reduction in predictability. And that chaos and the loss of predictability is much scarier than an increase in cost. And when I started talking to management, I started framing everything in terms of a risk-based decision. As in, “the deadline will be here either way.” And the decision is about where we want to be when that deadline gets here. It’s not about - as soon as you start negotiating things in terms of time, you get back to that metaphor of “How can we throw more people at the problem to make it go faster?”
And then the other thing I still see is people thinking about developers like a commodity resource. And what’s fascinating when you look at the effects of a loss of familiarity on the team, I have one case study in the book I did with this team that was an awesome team. But basically it was like post-exodus phase, where all the original developers left, had this huge breakdown. And then they got new management which brought in some great people. But they had this code base that nobody understood.
And the lack of familiarity cost - 80% to 90% of their time was spent figuring out what to do. And the cost when familiarity walks out the door is insane. I mean a lot of rewriting your software is just to reestablish familiarity, because we can’t work when we can’t relearn the system. And you just get stuck in that state of 80%, 90% friction. And I don’t think - we’re talking about interest rates - you don’t think about stuff that’s taking up 80% to 90% of your capacity. It’s unreal when you start looking at how much time we waste for all these problems.
Len: It’s interesting. You talk about technical debt, getting us to see as nouns what we should be seeing as painful verbs. I really like that, and I was wondering if you could explain - on the subject of technical debt, a little bit more about what you mean by that?
Janelle: Sure. This kind of goes back to the metaphor mapping thing. We’ve got different metaphors that we map in our brain. We’ve got objects, which are basically thing-patterns. We’ve got spatial relationships and context-type patterns. And then we’ve got process-patterns, or things that happen over time. And all these different metaphors have these three basic types.
One of the things I realized is that technical debt is a noun. And the effects of it being a noun mean that we look for things that are noun-like in our experience. And generally we look at, “Okay, so this part of the code is a noun - this technical debt thing. And this is a painful technical debt. This is a low pain or a high pain.” But it’s all one-dimensional. We bucket our experiences in terms of high-pain and low-pain technical debt. But idea flow is a process of understanding and extending the software. And so pain occurs in the context of a process.
And once we shift from a thing-pattern to a process-pattern, suddenly we start seeing our pain in terms of a journey. We’re in these situations. We make these decisions to navigate around obstacles, and have to dig through different areas of friction and break down constraints. And you start to see the problem solving experience, and the true complexity of what’s going on, because we can start relating to journey metaphors, or things that happen in time, as opposed to just one-dimensional things.
If you look at the patterns in my book, they’re patterns of mistakes and patterns of causes of troubleshooting time. They’re not like any patterns we have in our vocabulary at all. They’re patterns in development experience, that I didn’t start to discover until I’d shifted my foundational way. I started thinking about pain as a process, and what were all the factors along my problem-solving journey that ended up affecting my pain?
Len: I’ve got one last question about your book. I was wondering if you could say, for example, to someone who is listening is or has or will be managing a software project - how they can use the concept of idea flow to improve their work?
Janelle: So would this be for a developer or for like a manager?
Len: For a manager.
Janelle: Okay. One thing I’d say is that the techniques I’m proposing in my book need to be developer-owned, in that the development team needs to take responsibility for identifying their problems, understanding what they are, and figuring out how to fix those problems. Learning how to do that and discovering the right problems takes time. If we don’t allocate time to figure out what the right problems to solve are, then we’ll inadvertently end up solving the wrong problems whenever we try and work on improvement.
And if management can agree to let the development team work on the most important problems to solve - whatever those might be, and the development team agrees to gather data to identify what the biggest problems are, and then share the things that they’re learning with management - we just set aside capacity to do that, and that becomes the contract between management and development. Then you can start working your way to a better place, and then shifting that ownership of making those decisions and gathering requirements and solving those problems to engineering. And we can start steering our projects.
We talk about the importance of empowered teams. But it’s not just about self-organizing and figuring out what we ought to do, because we’re all too busy to do that. And so we just shortcut all the structural things, and do what we need to do right now - and worry about all the process stuff later. Because we’d much rather write code.
I think we need to have that structure. And it’s worth putting some time into designing our organizational structure - more like its architecture, and having that be a first class responsibility - of understanding the communication dynamics in our organizations. And start creating more human systems architectural roles, if you will.
I’ve also got some diagrams of modeling using software as a metaphor for human systems design, and some of the design problems in our organizational structure that cause pain. I think using that as a template, even if you do nothing else, and take absolutely nothing else from the book - that one change will make all the difference in the success of the project.
Len: Thanks very much for that. That’s a really great answer. I was just wondering what the next step is for you? You’ve got the book and you’ve got these great ideas that you want to spread, and I was wondering what your next steps are?
Janelle: Wow, I’ve got so many next steps. Right now, I’m doing a speaking tour with No Fluff Just Stuff. I’ve got a five talk lineup that I’m doing, breaking down these problems of how to do data driven software mastery. And then the organizational side, how to build a learning organization. And then my business, to get time to work on it - I’m a very firm believer in that you just get up and do it anyway. And you don’t grow without stepping outside your comfort zone. So I’m learning my way through it. I’ve had things that I wanted to share for a long time. And this book, too, took me five years to write, and rewriting things over and over again, and failing to communicate. Because all this stuff is really abstract. And I can iterate a lot faster on stage too, I’ve found. But it’s kind of a challenge.
But the thing I’m working on right now is building a community at Open Mastery around these ideas, which is an industry peer learning network focused on data-driven software mastery - essentially companies and individuals that are interested in trying out these ideas. I’m working on spinning up local communities, where we can start working on implementing these practices in our organizations iteratively, and then coming together in a local community group, and talk about lessons learned, and codifying what we learned into patterns and principles.
And then we’re building a new vocabulary of patterns on Wikipedia. And we’re hanging out and talking over Slack. And we’re writing blogs, … we have every day, to build a community around - but my goal is to lead an effort in learning how to learn together, with the pain on center stage. And there’s so many problems in our industry with broken feedback loops between the education system and our economic system, and between management and engineering.
If you start looking at a lot of the problems and pain in the world, it all kind of comes back to broken feedback loops. And so, what I’m trying to lead is an effort in learning how to learn together - as an organization, and as a community and as an industry. And hopefully, if I can get this going, as a species. So that’s kind of what I want to do with my life. And I’m hoping that I can find a group of people that are interested in learning along with me. Because it’s a whole lot of fun, but it’s a whole lot more fun to do it with other people.
Len: Thanks very much for taking some time in your life to do this interview. I hope it helps spread your ideas, and helps you achieve that. And I really hope you succeed, by the way.
I guess I’d like to ask you one question about Leanpub, and that’s, why you chose Leanpub for your book? Obviously, something that took five years to write, how you distribute it, and how you publish it, is a very important decision. I was wondering if you intend to keep it on Leanpub as permanently, or are you looking for a publishing contract?
Janelle: So your first question of why Leanpub - I mean, being in software development and Agile and lean being so central to my life, I live and breathe “iterate”. And having a platform that was really easy to get started with, that I couldn’t get in my own way with all the excuses I would make of not doing this is just like - I remember first reading the Leanpub documentation, and how writing your book in Markdown without all these formatting things was a feature.
And at first I was upset that I didn’t have all these fancy tools. And then I started to realize, “Oh, I get what this means now,” of how much I get in my own way from writing my book. Because I’m sitting there fussing with what things will look like, as opposed to just writing the material. And I came to really like the flow and the way that the tools are organized in your content’s organized - it’s just really straightforward and easy to work with. And you focus on writing your book, as opposed to anything else. I’ve very much enjoyed that aspect of the experience. It’s been great. And then I can easily publish drafts and get feedback on things, which I very much needed. My book wouldn’t be anywhere near what it is today without five years of feedback. And some of it not the nicest feedback that you want to hear, but it doesn’t mean that it’s not what you need to hear. But yeah, I’ve been thankful for the tooling, and it’s awesome.
Len: Cool, thanks. I guess I have a tendency sometimes to ask two questions at once, but are you going to be looking for a conventional publishing contract and distributing the book through a conventional publisher at some point?
Janelle: I am. I’m working with O’Reilly right now, so I ended up taking my book and splitting it into two, because I just had way too much content. So what I’m working on right now is - I’m like 80% of the way through a second book, for which my tentative title is, How to Build a Learning Organization, which is kind of the learning framework side of Idea Flow. I ended up focusing Idea Flow just on how do you measure pain? And then the learning framework itself, I’m splitting into another book.
So one of the things that I’ve learned in trying to sell these ideas, is that I’ve got a two-sided audience, one of them being developers, and the other one being management. And so my second book, I’m going to focus more on looking at the problem from a top-down standpoint of what managers need to do to support engineers, and what does the organizational protocol look like from each side of the organization? And then, how do we start building an integrated organizational learning process, so we can create a steering wheel of sorts at the organizational level - which is a hugely complex problem.
But I’ve got an initial set of blueprints as a starting point, and then I’m using Open Mastery as a vehicle to get the community to take ownership of getting from point A to point Awesome. And so, let’s see. What was your original question, I forgot.
Len: It was, are you going to be looking for a conventional publishing contract?
Janelle: Oh, yeah. So I’m working with O’Reilly, and I’ll keep my book on Leanpub probably as long as I can. But my main goal is trying to get my book in front of as many eyeballs as possible, more than anything else. And working in partnership with a giant marketing machine has its advantages.
Len: I just wanted to say on that note, for us at Leanpub, when someone starts their book on Leanpub, and then ends up going in the end with a publisher, and retiring their book on Leanpub, almost nothing makes us happier. I mean that is just a fantastic outcome for the author and for us, because part of our goal is to open up that time when a book is normally hidden in stealth mode on a person’s hard drive, to the public and to people.
If all books were published while they were progress, and then get taken up by publishers at the end, it’s a win on all sides for everyone, including readers and authors.
I guess my last question is selfish. I mean, speaking to someone who thinks so much about improving processes, if there were one thing that you could ask us build or to improve about Leanpub, what would that be?
Janelle: Probably the thing that I feel like I’m missing the most, that I’ve been trying to make up for in other kinds of ways, is the ability to build a community around the book, that’s integrated with the tooling. I know there’s kind of email list setup, but when people buy the book - unless they check the little box of share their email. it’s really hard to create relationships out of that. And I’ve tried to come up with some other ways to work around that with signups through other kinds of tools to collect emails, so I could stay in touch with the community. But that’s probably the big thing I feel like is in my way as an author is, building, getting the community of people around the book talking to each other, as opposed to just this one-way communication. And I realize there’s discussion comments and stuff, but it just has a very different kind of feel to it.
Len: Thanks very much for that. You’re the second author I’ve interviewed just in the last couple of months to talk about community when asked that question. So that’s a very strong signal, and it’s actually something that we’ve been thinking really hard about. Hopefully, somewhere in the near to medium term, we’ll have a better community experience. As you noted, we have a sort of double-blind “email the author” process if people want to use it that way. And we’ve got Disqus comments if authors don’t turn it off.
But there’s just so much more we know we can do around that, including from our perspective, we’re sort of building the world’s first library of in-progress books. And what does that look like? And what is the importance of community around that? There’s lots of interesting questions around, for example, reviews, right? Back to 1996 Amazon. But what do you do with a review of an unfinished book, when a new chapter is added, or something like that? There’s an interesting aspect of time that’s involved. And it’s just so interesting to think around building community throughout that time, and around an unfinished thing.
Janelle: Yeah, I can see that being a really difficult challenge. I think about some of the early feedback I got that was just awful feedback - stuff that you’d never want to make public. Like, “I have no idea what this book’s about, or what you’re trying to say.” It was like, “You just need to rewrite this whole thing from scratch. It makes no sense.” I mean, feedback like that, you don’t really want to necessarily be a big public spectacle that you can never get rid of. I think there needs to be some privacy around certain aspects of feedback, especially - I don’t think you’ll get the kind of feedback you need in a lot of cases unless it’s private.
Len: That’s a really good point.
Janelle: It’s like there’s two different– Yeah, I’m not sure. It’s an interesting challenge you guys have.
Len: Well, that’s a really interesting. I hadn’t thought of that before. Because yeah - one of the processes that’s pretty commonly used by Leanpub authors, is to put their email address in the introduction to their book, and say “Please email me.” And then that is almost always a private communication. But I hadn’t put it together, that one of the reasons that’s actually - it sounds so clunky - but the reason it works so well is that it has this value of being private.
Len: That’s really interesting.
Well, I think this is my first feature length interview! I want to thank you very much for your time, and for all your great answers to questions - and for your book as well. So, thanks very much for being on the Leanpub Podcast and for being a Leanpub author!
Janelle: Thank you for inviting me. This was really fun.
published Jul 20, 2016
Nicholas Russo is a network professional who has earned CCIE certifications in both Routing and Switching and Service Provider. A former Comm Officer in the US Marines Corps, Nick is also the author of the Leanpub book CCIE Service Provider Version 4 Written and Lab Exam Comprehensive Guide. In this interview, Leanpub co-founder Len Epp talks with Nick about his career, the duties of a Comm Officer on deployment in wartime, and about his book and his experience self-publishing on Leanpub.
Please note that the views and opinions expressed by Nick are his own, and do not represent the views of his employer, Cisco.
This interview was recorded on June 29, 2016.
This interview has been edited for conciseness and clarity.
Len: Hi, I’m Len Epp from Leanpub, and in this Leanpub Podcast, I’ll be interviewing Nicholas Russo. Nick is based in the state of Maryland in the United States, and holds a Bachelors of Science in Computer Science and a minor in International relations from the Rochester Institute of Technology. Nick is a former Comm Officer in the Marine Corps, and recently he was one of the first people that passed the Cisco Certified Internetwork Expert, or CCIE, Service Provider, Version 4 Lab Examination.
Nick is the author of the Leanpub book, CCIE Service Provider Version 4 Written and Lab Exam Comprehensive Guide. His book is about achieving certification in the CCIE Service Provider track, which has recently been updated by Cisco to reflect changes in standards, techniques and procedures made by networking vendors. The book is meant not only for people interested in passing the related written and lab exams, but also for enterprise architects and other network professionals interested in the core routing concepts for any network.
In this interview, we’re going to talk about Nick’s career, his professional interests, his book, and at the end we’ll talk about his experience writing the book, and self-publishing on Leanpub. So thank you Nick for being on the Leanpub Podcast.
Nick: Thanks Len, I’m happy to be here.
Len: I usually like to start these interviews by asking people for their origin story. So, I was wondering if you could tell me how you first became interested in computer science, and eventually made your way to the Marine Corps and on to Cisco.
Nick: They had a relatively new computer science program when I started high school in the year 2000, and they had a bunch of new computers. I didn’t really know what computer science was, but I knew I liked computers as a kid. So I figured, “Okay, well I’ll go check this out”. And then I quickly found out that it wasn’t just putting cards in computers and playing games. It was about learning how to command computers to do things, and to solve real life problems.
I immediately saw value in that as a high school student, and I managed to do those courses for all four years of high school. That led me into the college degree of Computer Science, when I went to RIT to pursue that career. Along the way, in-between college, in between high school and college, I joined the Marine Corps in the Reserves. Partially - mostly for the adventure of it. I didn’t really have a great reason at the time, but it’s something I knew I wanted to do.
So I did the Reserves while I was in college studying Computer Science. While in the Reserves, interestingly, my job had nothing to do with communications at the time. I was simply an infantryman - like the bazooka guy, basically - which was a fun job, but not something you really want to take too far. Once I got done with college, I thought about that, and said, “I’m going in the Marine Corps, I’m getting commissioned as an Officer”.
When I went to training, at that point, I got to decide which way do I want to go. Do I want to continue to be an infantryman, or do I want to actually put my computer science skills, at least to somewhat of a use, and go into telecommunications? So I decided to go that route. On my second deployment to Afghanistan, which was in 2011, I got really interested in networking.
This was mostly the result of one of my fellow Marines. His name was Master Gunnery Seargent John Robertson, who’s one of my best friends, one of the most influential and smartest guys I’ve ever worked with. He could make anything work out of nothing. You see a lot of these stories, it’s kind of a different form of innovation, if you will. A lot of cool stuff comes out of Silicon Valley these days, people solving a lot of real world problems with new ideas.
This was the kind of “guy stuck on an island” innovation, where I have a nail and a shoestring, and I need to communicate 20 miles. It was that kind of stuff. I really enjoyed that, and I could see a lot of the ingenuity that came from the tactical side of communications. But then I became really interested in the large scale routing, switching and service provider technologies that were a little bit related to what we were doing in the tactical space.
When I left the Marine Corps, I worked for a company called Harris RF Communications in Rochester. They had a very large tactical radio program, networking tactical radio. So it actually worked really well. My experience with radios in the military, plus a little bit of my network knowledge, it was a good way of bridging that gap, where I could connect radios to a network, and extend my IP connectivity over a tactical radio net, so a guy driving around in a truck could access different network resources.
I moved on past that a couple of years later. Ultimately I went through a couple of other jobs, and now I work for Cisco as a Network Consulting Engineer - doing a lot of large scale routing designs and support for the Tactical Army, as well as other army garrison posts.
Len: That’s a great story. I think most of the people who listen to the Leanpub Podcast are still currently computer programmers, and I think the kind of work you described as a Comm Officer on deployment is something they might like to hear a little bit more about. Can you give me an example of what a typical day might have been, and what a typical task that you might be assigned to do would be?
Nick: What’s interesting about it is, computer science and networking - they’re different in many ways, but similar in others. The thing about computer science is you’re trying to take a machine that has no brains and give it brains, so that it can do something - solve some problem, generate revenue for a company. Or even something like, “Hey, sort this list of items using some kind of algorithm”.
In the Marine Corps, doing communication stuff, we have a whole bunch of items which are done - these radios have software on them, these routers have software on them. So the software engineering component isn’t really in play. However the systems integration component and the networking piece was really the major challenge there.
And as you can imagine, when you’re in a combat zone, your resources are limited. In addition to human factors, like stress and personnel shortages, you’re asked to do things like, “Hey, we need to communicate to these new sites. We’ve got these new outposts. These guys need connectivity”. That’s an important task, and I don’t have the resources either in people or equipment to do that.
So then you start to be creative with the tactical radio thing I was talking about. Like Harris for example, they had something, “Hey we can drop these small little [Manpack sized radios] out there, and extend IP connectivity to these guys, because we don’t have a giant truck that we can send out there for satellite communications”. So things like that, I think were really interesting.
And of course, you can’t just drop the radio in some guy’s lap and say, “Okay, turn it on”. That’s not going to work. There’s a whole planning detail that goes into that. Both from a spectrum perspective, making sure that you’re allowed to use the frequencies of that radio, as well as the configuration of the network side of it.
So doing that, I would say - on an interesting day, where we got to do something like set up a new site, or deploy some new services to some outposts - we would typically find ourselves trying to solve those kinds of problems. Ultimately, you think about it from a business perspective. That the business value is, we’re enabling a new site. Which is at the edge of our perimeter, ultimately providing security for the rest of the unit.
Len: I imagine that setting up a network so that people can communicate with their families back home, must be a really important part of the job now as well?
Nick: Yeah, that was another interesting part of the job, because typically in a tactical network, when you’re doing these networking things and enabling services for different units, you’re typically focused, number one, on the classified network. Because that’s where you exchange all your fire missions and that’s where you fight your battles. After that comes the unclassified tactical network, where you’re going to do things like logistical coordination and things that aren’t classified, but still protected. And then the last, and lowest priority of course is the morale network.
We were fortunate enough to actually have a pretty decent morale network. Mostly because the first two priorities were pretty solid. So we had a little bit of a morale station if you will. People would come by the main FOB, maybe on a monthly or every couple of months basis. We had a couple of computers set up in the corner where people could - it’s basically an internet cafe for free.
People show up, they hop on to check email, they do a video call, and then they go about the next couple of weeks being without it. So that was a pretty good thing to have as well. Obviously for morale purposes, and also because - it’s another network to maintain, so then you start to get into this habit of, “I have multiple different security enclaves, I need to be able to maintain all those different things”.
Len: You mentioned FOB, which I understand is Forward Operating Base.
Nick: Yeah, that’s right.
Len: Would you actually go yourself to FOBs?
Nick: I did a little bit of that. In my role as a Communications Officer, I was a little bit tethered by some of the leadership. Because they’re like, “Hey, we need you - you’re the bellybutton for us. When we need communications, we come and poke you”. So it was often times a lot of my subordinate leadership again - John Robertson was one of the instrumental people - he personally did a lot of that himself, and also helped me delegate that to some subordinate leadership within my unit that I was in charge of.
But what’s interesting is, the chances that I did get to go out there were kind of a real eye opener, because sometimes when you’re sitting behind a desk, you’re like, “Hey, you guys go out and set this system up, and it needs to be up by tomorrow”, and when it’s not up by tomorrow, you start to get upset. You’re like, “Oh come on, I needed that up now”. And when you go out there and do it yourself, it’s like, “Oh, that actually wasn’t so easy”.
That’s where I started to gain the appreciation for networking. It’s like - I sit here, and I think their stuff is easier than it really is, because I’m looking at a diagram with some routers and switches, and I just have to draw a line - and my job is done. Then I realise it’s never like that.
Len: It’s really fascinating. I was thinking as I was going over your book about the routing and switching part of computing. I imagine this might kind of annoy you at some point, but it’s one of those things that everybody who uses computers and communications devices just takes for granted is going to work. It’s the black box underneath it all.
I was wondering, if you were to explain, say to someone non-technical, what it is that you do - just briefly, how would you do that? And explain what the importance is of the technology that you work on? Now at Cisco, I guess.
Nick: Sure. So my job here at Cisco, it’s similar to what we kind of already do. You can think about my old role when I was in the Marine Corps. I was a Communications Officer in an infantry battalion. I supported communications for that battalion. Actually I was the operations guy, in a way - the Marine Corps fielded me all this equipment. They gave me 10 of these com vehicles, we’ll say, and I’ve got to supply reachability to 12 units. Okay, that math doesn’t work out so easily, so then it’s up to you to think of ways to solve that problem uniquely.
The role I’m in now is, I work to support one of the program officers that actually builds those trucks. So I support them in large scale networking. Because from their view, they’re fielding hundreds and thousands of these trucks across the entire US Army. So you can imagine the network that it takes, and time and resources - what it takes to maintain, plan, operate a network like that. It’s pretty substantial. So I work on that quite a bit to help them determine, What kind of architecture do we want done? How can we make this scale? How can we make it manageable?
And then the other important thing - the people running this network are soldiers. And nothing on them - they’re extremely courageous, smart, tough guys and girls. But a lot of them, they’re not professional engineers a lot of the time. So we always have to consider, this has to be real easy. Kind of like smartphones and all these new smart devices we have. Anyone can use them, but the tech inside of that thing is just overwhelmingly awesome I think. It’s crazy, it’s complicated, but it’s awesome. And somebody had the brilliance to take all that tech, and put such a simple interface on it and make it resilient. And that’s one of the biggest challenges that we have in tactical.
Len: I was wondering, on the subject of technology, what would some of the recent developments be that have had to be incorporated into your engineering processes?
Nick: Some of the recent developments - I mean, we try to always keep a thumb on a lot of the recent developments that happen within the industry. So for example you hear terms like SD-WAN and Software-defined networking, and just general terms like that, like cloud computing. We try at a minimum think through how we would want to deploy these things in a tactical environment.
I think there’s a lot of promise for those technologies, specifically things like SD-WAN, where I might have slow satellite links and fast line-of-sight links. Of course, all these things are built on the fly, and change very constantly. So some kind of intelligent system that can route traffic, based on different metrics and other user-defined criteria, would be extremely important. We’ve continued to keep a pulse on things like that.
The other thing that’s always fun in a government environment is, appropriating money to buy these things, field them, test them. That cycle can sometimes be very long. And you think about small startups, a smaller company of a few guys, they can make a couple of decisions and change the direction of their company incredibly over the course of a few hours, where a bigger ship is always harder to turn.
Len: On the subject of technology, drones and UAV’s are obviously on the top of many people’s minds who read tech news. I was wondering if that’s part of the communications infrastructure that you’ve worked on?
Nick: It is a little bit. We’ve talked about having kind of like an aerial tier in a sense. So you can kind of imagine, I’ve got a bunch of trucks on the ground, and maybe those trucks can’t see one another. But satcom bandwidth can be very expensive, and sometimes prohibitive. So if I can have some kind of aerial drone to serve as almost a re-transmission or a relay point - we’ve considered things like that as well.
Those things are continuously on our radar, because deploying capabilities like that, both obviously from a surveillance perspective, for intel feeds - which has been around for a long time - but also for radio re-transmission systems. Those are highly relevant for tactical.
Len: On the subject of your book, as I understand it, at the time you passed this latest certification, you were one of only 10 people or so in the world to have done so. You can correct me if that’s wrong! I just wanted to ask what some of the changes are that are reflected in the latest version that required a new certification test?
Nick: That number I think is kind of a scientific guess - a wild guess based on what I’ve talked to a couple of other people in the group about. Long story short is, every couple of years, most vendors, when they offer a certification test, they’ll come up with, “Okay, we’ve got all these new features that are relevant in industry. We want to make sure that we certify people on these topics”. But then after three, four, five years some of those things may lose their shelf life, they may not be relevant. So vendors try to do a good job, and cycle those things out.
The CCIE Service Provider version three to version four, at least in my opinion, had a relatively sizeable number of things that were added and only a small number of things that were removed. So ultimately, what you end up with is a very, very large topic domain. And that’s what drove me to write the book.
What’s interesting is, for a test like this, there’s training materials out there, there’s documentation. There’s a lot of information. There’s people to talk to. But a lot of times what’s really hard to find is a consolidated set of labs and technical explanations, to a sufficient level of depth that would allow you to succeed in passing the exam. There are companies that exist out there - very good ones in fact - that base almost their entire business model on the CCIE and passing those exams. And they’ll have professional instructors that do videos, they build labs, they have workbooks, they have training sessions. It’s a massive - I wouldn’t say massive, but it’s a sizable industry.
Obviously my book doesn’t add the human element, per se. I’m not doing boot camps and stuff like that. But I do cover all the topics in the blueprint, and a lot of people have contacted me looking for materials. And of course, now that I’m finally able to offer something, I think that there’s a lot of interest in it.
I think just in general, it’s kind of a human thing - “Hey I really want to achieve this thing, but I don’t have guidance, I don’t have a mentor, I don’t have information. I just don’t know what to do.”
So people typically do one of three things really. Either number one, they wait for it. They wait for someone to do it. Which is fine, and basically all those people are my customers now, and I’m more than happy to support them now that they have my book. They waited a couple of months, they got something.
The other group of people are people who just quit. And that’s totally understandable. “I don’t have materials out there, I know I’m not going to succeed”. Maybe they’ll try to go for a different task, maybe they’ll go back to college instead; maybe they’ll spend that time with their family, instead of studying.
And then the third type, which I think would be me, is the guy who goes and writes his own book on it, in lieu of not having any other materials. And that obviously takes a tremendous amount of work. I know you see the page count is almost 3,000 pages. There’s almost a million words in this book. It took me about seven months and 1,000 hours of continuous dedication to work through, to document these technologies and verify everything several times, so that I had a very clear understanding of what was in it.
Because my logic is, if I’m going to try to market this to my customers, I want to ensure I’m providing them a quality workbook, that’s going to accurately reflect the topics on the blueprint, as well as all the ins and outs of the weirdness. Because of course, when you deploy technology, it never works the way you think it’s going to work. So you always want to have that explanation of how you’re doing things, and especially why you’re doing things.
Len: You mentioned in a blog post that you actually failed the first time you took the test. I was wondering if you could talk a little bit about what you learned from that experience, and helped you pass the next time?
Nick: Yeah, that’s the interesting thing about these exams. I think they say the second attempt is common, because it definitely is. I took another CCIE in 2014, the Routing and Switching test, which I also failed once. I passed that on my second try, like this one. And taking it the first time, I went in there highly confident, my book was already done.
I’d already written the whole book, I knew everything in the book. I was at the top of my game. I walked in, I got through the first couple of sections pretty well. And some things later on in the test, I found extremely difficult. I was pressed on time, I was tired, I was hungry. All those human elements start to eat away at you as you’re trying to accomplish this very difficult task in a very limited amount of time. And at that time, the time I first took the test, I didn’t know a single soul. I’m sure there were some in the world, but I didn’t know a single person that had passed the test.
By the time I passed it the second time, I knew about two or three people [also had]. So it was heating up a little bit. I’m sure to date there might be 20 or 30, but it’s still a relatively small number. And I think the reason for that small number, it’s not so much because of the impossibility or the difficulty of the test. I think it comes back to the fact that people just don’t have the information that they need to pass it. And I’m trying to help break that, so I can bring some more knowledge to people, and ultimately help raise the popularity of people achieving this exam.
I know there’s a lot of people who use these technologies, and use similar technologies in their everyday work. People who work for large service providers, the large tier one carriers in the States. You’d think of Sprint and CenturyLink and AT&T. These guys have massive, massive carrier networks, and a lot of the technologies in this book are very, very specific to that environment - and also for large enterprises as well. In my opinion, having a mastery of those technologies is how you build large scale networks that are scalable and efficient.
Len: I noted that you talk in the book, I think in the introduction, you say that the book is obviously useful not only for people who are taking this specific test, but for people who want to be experts, or are experts in their field, and just need information about this somewhat undocumented subject.
One detail I liked from your blog post was, you mention it’s good to read - and I think this applies not only to the exam the book is about - you talk about how it’s important to read the entire exam, all the questions at the start. I was wondering if you could talk a little bit about that. Is that so you can prioritize questions? How would you go about prioritizing them if that’s the reason?
Nick: Yeah, that’s a great question. The thing that’s interesting about these tests is, and a lot of people will say this, and I tend to agree with it: a lot of getting through that exam is the strategy and how you go about it. Like I said, when I took my first CCIE, I failed it the first time, but I knew all the technology. I was extremely competent. Same thing when I failed the service provider test the first time - I knew all the technology, the book was done. I was extremely fast on the command line, I knew what I was doing. But it was a strategy thing that I didn’t have nailed down. And the thing about the strategy, as you know - you’re asked to design a network either from scratch, or you have some kind of a brownfield deployment.
For example, you have a network that already exists, and you’re asked to make significant changes to it in an extremely short amount of time. So you have to look at it and say, “What does the network look like now? What are all the things they’re asking me to do, and how can I put these tasks in the proper order?” So it’s both a prioritization thing, like, How can I get the most points as fast as possible? But it’s also a matter of which tasks make sense to do in which order, because sometimes what you might end up with is, in the beginning they have you do some basic routing tasks, and then in the middle of it, they have you do a bunch of value added service stuff. Maybe like IP multicast or quality of service or MPLS Traffic Engineering.
And those things are important. You’ve to do them to get the points. But they might have you come back and do some routing stuff later in the test, when it would’ve made more sense to do that upfront. So you kind have to think about, What is the overall objective? What does the network need to look like when I walk out of this room today? And that’s what I need to focus on from the very beginning.
Len: I was wondering if you could talk a little bit about why you chose to self-publish the book, and why you chose Leanpub as your platform?
Nick: The self-publishing thing, it’s - I’ll keep this brief, just in the interest of not boring everyone. But I wrote this book, and I finished it a couple of days before I joined Cisco. And because I did that, I talked to my company, and we worked it out and said, “Hey, I wrote this book before I came here, but now I work here and I’m trying to publish it”.
Typically when you work for a company, what you develop at that company is property of that company. So we went through a little bit of a thing there. Everything was great, they were extremely cooperative. It just took a little bit of time to get through the approval process. And then at that point, I was like - well, it’s June, I wrote this book in January and there are people failing this test around the world.
The test was introduced in 2015, and I figure the test has about three more years of shelf life on it. If I spend another six months working with a professional editor, it’s going to cost me a lot of time and a lot of money. And quite frankly, a book with a million words, it’s going to have some grammatical errors, it’s going to have some spelling errors. But it’s a technical book, and all the content is accurate. I think that’s what’s really going to matter for my customers, and those who have put their testimonials up on the website will certainly agree that the content is certainly there.
I figured I would rather push this out now, get it published quickly on a place that I think is reputable, and people are familiar with. And especially Leanpub, I think a lot of the information - I look at a lot of the books on Leanpub, they’re written by people who are coming up with new ideas about software and APIs and things like that. And it seemed like a very - I guess, for lack of a better term - a very tech-friendly environment, where I could post something that I came up with, and it would get viewed by people with similar views, and people who would potentially be interested in it - even if they’ve never heard of it before.
Len: Thanks very much for that, and for choosing Leanpub. Your case is one of the reasons we built Leanpub - so that people who are, say, on the cutting edge of something, and an expert in something, who’ve made something that has a kind of time sensitivity to it. Also, something that’s so useful to the people who need it, that the occasional grammatical error, or lack formatting finesse is actually irrelevant to them.
This is actually an example I’ve often used, just sort of in general. If you approach a publisher and say, “There are a thousand people in the world who are expert enough to understand my content, or to be interested in it in the first place”, You’re pretty unlikely to get a publisher unless you’re an academic, or you’re going to an academic press or something like that.
But in your case, you can publish a book, and provide a lot of value to your customers, but also get something back yourself. And because we pay a 90% royalty [minus 50 cents per transaction - eds.], then it can be worth it, even if you in the end you don’t reach a great number of readers.
Your book is really interesting in a very specific way, because, as you say, it’s nearly 3,000 pages, nearly a million words, and so you’ve set the minimum price at $200 and the suggested price at $300. I just want to explain to everyone listening, one of the reasons this is so interesting, is that Leanpub books have no digital rights management or DRM on them, which means technically you can make as many copies of them as you want and distribute them. And you can also get a refund with just a couple of clicks, no questions asked.
Your book so far I think has one return or something like that. So people are voluntarily paying, obviously, is my point, and I was wondering how did you decide to set the minimum price to $200? Was it based on instinct or some research?
Nick: The thing that’s interesting is I checked a couple of competitors out there. And just to be clear, I looked at some of the competitors who were offering workbooks and despite some of those other workbooks being much, much, shorter - like on the order of 600 to 800 pages, and being relatively incomplete, those books cost about $200. So I figured for my book to be a little bit more expensive, that’s appropriate. But for them to be the same price, there’s no contest. At least that’s the message that I’m trying to put out there. You can buy a 3,000 page book that’s known to be complete for price X. Or you can buy another book that is a quarter of the length for the same price. And you can choose based on that.
The $300 price was my initial idea for it, but I figured if I’m going to launch this book - at least for me personally, I think getting the exposure as an expert in this field, and this is my first book that I’ve ever published - being able to do that, and to call myself a professional writer with a publication, and to put it out there - in my eyes was more important than making that extra 50% on the royalties by selling it for 100 bucks more.
I think what’s interesting is, I didn’t really expect to see a whole lot of people voluntarily paying more. But there actually have been several, which is very cool. People I think that have said, “Hey this book is 3,000 pages, and if I spend 300 bucks, that’s 10 cents a page”. And you can imagine the amount of technical detail you’re getting for just that one 10 cents. You’re getting a whole page worth of extremely detailed stuff and explanations. Entire topics on here might be anywhere from 10 to 100 pages. So for a dollar to $10, you’re getting an entire topic that describes a technology that may not be documented sufficiently in any other study guide.
Len: You mentioned in an email the other day that your book is already up on torrent sites. Every author loves their books and puts a lot of work into them, but yours in particular is special in that regard. Given that you’re a first-time author, I was wondering just how it feels to have your book put up on torrent sites, and how you’re choosing to mentally, or even procedurally, deal with that?
Nick: It’s one of the things to be expected in the certification world. This is one of the the warts on certifications in general. With enough Googling and with enough searching on the net, you can find answers to probably any automated test out there. And that’s just really unfortunate. Typically when you meet guys that kind of cheat their way through these tests, you can see through them like glass. But nonetheless, I like sharing information. I’m certainly willing to do that. I feel like I’ve set a fair price for the book.
And the thing that I also believe, and this is actually, I kind of stole this from one of Cisco’s values - one of the things we say in Cisco is, “Intensely focus on your customers”. I do that at work with my Cisco customers of course, but I also decided to make that kind of the flagship of my publishing career. I get personal emails from a lot of my customers who use the email link on Leanpub, after they bought the book, to ask me a question about, “Hey on page 2,000, I have a question about this command. How does it work?” And I make every effort to answer that question within a couple of hours. I’ve already taken on to try and mentor - some people have asked me to help me mentor them through their study journey. I’m willing to do all that for anyone who wants to get on board with the book, ask questions about it.
I think that’s extremely important to being successful, basically in whatever you do. I’ve heard a really great saying, “When you focus on making money, you ultimately fail. But when you focus on your customers, you ultimately make money”. So even though making money’s not really the goal here, I just - getting the exposure, becoming a professional writer, publishing a book and knowing that I had the spine to go through and write this thing and publish it and take it from cradle to grave - I think brings me a lot more benefit than the money.
Regarding the torrent-site stuff. I’m kind of half-hearted chasing that down, in terms of infringement, things like that. I don’t want to make that the cornerstone of what I do every morning when I wake up and I roll out of bed, and I start searching torrent sites. I’ve got better things to do with my life. I’ve got more books I want to write. I’ve got customers that I want to help get through this test, and I think that’s the most important thing to stay focused on.
Len: That’s great, both of those answers I would say are also textbook Leanpub principles, but expressed probably much better than we’ve ever managed to express them ourselves in the past. And we like trying to do that. So thanks very much for that. I mean, engaging with customers, and focusing on the kind of work that you as an author ought to be doing, are two things that are central to what we’re trying to deliver for people as well.
One specific question I have regarding engaging with readers and feedback through email and things like that is, Do you plan to incorporate any changes that people suggest to you, or corrections that they find into your book, and publish new versions?
Nick: That’s a very interesting question. I want to say yes, but I must say no. And the reason I must say no is - as I mentioned, I [finished] this book about a week before I joined Cisco, and anything I do now today would be a Cisco thing. However, I’m working very closely with the service provider program manager for the CCIE. She and I talk probably on a weekly basis, and I told her, any errata to the book, if people want to see other technologies, maybe something I’ve missed or - if there’s any corrections or supplements to do, I’m going to try to work with her and just publish official white papers from Cisco that try to cover those gaps, while at the same time, improving the product that the PM is offering, additional full scale practice labs and other things that Cisco is going to be able to use to help people get through this exam.
Because now that I wrote this book outside of Cisco, and I can collect that money not related to Cisco, at the same time, the real goal is to continue to keep people interested in this track, and assist Cisco in doing that as well. So now I’ve transitioned from, “Hey, I’ve published my own book, I’m running that on the side, but I also want to transition to assisting the Cisco Program Management Office, who runs the CCIE service provider program, providing them ideas for labs, and simply helping them actually build the labs - as property of Cisco. And it’s something that they can either market or give away, or build in their own study group.
So I would definitely encourage any feedback and corrections. Please send those to me, I’m happy to consider those kinds of changes for any kind of future documents that we publish for Cisco.
But the thing that’s interesting though is, the one thing that’s a little bit orthogonal to the way Leanpub works - “Publish Early, Publish Often” - well for me, it’s “Publish Once”. Which is a little bit odd. Ideally I would go back and fix the problems, but changing a - if I change a period to a comma, then I kind of enter a gray zone with legal stuff. And at this point, it’s probably not worth doing.
Len: I just have one more question, which is, we take customer development very seriously at Leanpub, and now that I’ve got you here, I wanted to ask, If there was any one feature we could add, or one problem we could solve for you, what would you ask for?
Nick: I was thinking about this yesterday. This kind of goes back to that one return. Again, I don’t think one return out of 35 sales is anything to be concerned about. However, the thing I think is interesting - the two-click return policy, I think is great. I think what would be potentially beneficial is two clicks and a mandatory drop down that maybe just says, “Hey, pick a reason. You don’t have to comment, but just give me one out of these five reasons - I over-drafted my account or the book wasn’t what I thought it was”. Or, “I bought the book, but I couldn’t download it for four hours and I got frustrated and I want my money back.” Just a basic reason. You don’t have to justify it, Leanpub isn’t going to grill you on it. The author’s not going to grill you on it. Just something for a record.
I think that’s valuable, because if the number of returns starts to become significant - for example if I see 10 returns in a week, and they all have the reason of, “I couldn’t download the book”, that might point towards a technical problem. Or it might be something like, “Hey, we all bought this book, and it wasn’t what we thought it was”.
Well, maybe those 10 people are part of the same organisation, and they were briefed by someone - maybe a friend of mine - a third party, “Hey you guys need to check out this book, it’s great”. Maybe he misspoke, and set expectations incorrectly. And now we ended up with a bunch of returns.
So just having that reason I think would be beneficial. And then of course a comments box, so that people could expand on that if they want to. That would help authors I think, just to understand what are the big muscle movements that justified the return. I think there could be some value in that.
Len: Thanks very much for that suggestion. Just for our listeners, what Nick is referring to is our two-click, 100% Happiness Guarantee return policy, which lets people easily get returns on Leanpub purchases. When they do that, they’re prompted to optionally leave a comment. We don’t ask for feedback in any other way. And we don’t require a comment either.
And that means that sometimes authors like Nick end up with mystery returns, where someone returns the book without saying why. And that can be unsettling. I’ll talk to my co-founder Peter about your suggestion, because I think it’s very good and could really help authors a lot, and maybe even help readers feel like they’re participating in a process at a deeper level as well.
One comment I would make in my experience watching Leanpub books and authors and returns over the years, is that usually if there is a problem with content, one or two people might return it without commenting, but if there is a real problem, someone will eventually comment. And often what they’ll do, if they understand how Leanpub works - I mean in your case this wouldn’t be the case, because you can’t change the book - but instead of returning it, they’ll contact the author through our “Email the Author” form, or on Twitter, and say, “Hey, can you fix this problem on page five?” Or something like that. And then the author will do it, and update the book and let the reader know. And then the reader feels like they’re part of the process, and it’s all a very happy thing…
Just before we go, is there anything else you wanted to say or an issue you’d like to address?
Nick: No, I just want to, I’ll just thank Jeremy Filliben. He’s a Cisco Certified Design Expert, or a CCDE trainer. He’s out of Delaware in the United States. He was actually the one who introduced me to Leanpub.
He and I were chatting on Slack one day. This was about a month and a half ago. I was complaining to all my friends I was having issues with publishing, I was having issues with the book. After writing this big book, I was right at the finish line, just trying to crawl across and get my book out there. And I was having a lot of different non-technical personal related issues with people and process and things like that.
He says, “Hey you should check out Leanpub”. I’d never even heard of it. I went to the website, I read through how does it work. And I read the FAQ page. And I was like, “Yeah, these guys look pretty cool”. And then, when I was approved to do the book and I was ready to go, I had the book in hand, and I said, “Okay, I’ve got multiple different self-publishing options here”. I kind of weighed those out, I talked to Jeremy again, and ultimately went with Leanpub for my publishing.
Len: Okay, well thanks very much Nick, good luck with the book! Thanks for being on the Leanpub Podcast, and for being a Leanpub author.
Nick: Alright, my pleasure. Thanks Len, I appreciate it.
published Jul 07, 2016
Amir Rajan is an indie game developer based in Dallas who is best known for having made the bestselling iOS app for the popular text-based game A Dark Room. He’s also the author of the popular and aptly-titled Leanpub book Surviving the App Store. In this interview, Leanpub co-founder Len Epp talks with Amir about his career, the experience of writing a bestselling iOS app, about his book, and about self-publishing on Leanpub.
This interview was recorded on April 27, 2016.
This interview has been edited for conciseness and clarity.
Len: Hi, I’m Len Epp from Leanpub, and in this Leanpub Podcast, I’ll be interviewing Amir Rajan. Amir is a Dallas, Texas based indie game developer and independent software consultant who’s experienced working with a number of platforms and languages. Amir also works on several open source projects, and speaks at conferences and user groups, and he blogs at amirrajan.net.
He’s best known for developing the iOS app) for the popular text based game, A Dark Room, which was originally created for the browser by Micheal Townsend (Amir also released an Android version recently: A Dark Room - eds.). Amir’s app hit number one in the App Store, and it’s been downloaded over 2.5 million times. You can follow him on Twitter @amirrajan.
Amir is the author of the Leanpub book, Surviving the App Store: How to Succeed as an Indie Developer, which is a very well-titled book. It’s full of lots of great tips, a very practical and valuable approach to helping people get started in game development for the App Store.
In this interview, we’re going to talk about Amir’s professional interests, his books, his experiences using Leanpub, and ways we can improve Leanpub for him and for other authors at the very end. So thanks, Amir, for being on the Leanpub Podcast.
Amir: No problem, it’s a pleasure.
Len: I always start these interviews by asking people for their origin story, so I was wondering if you could tell me how you first got interested in coding and being a developer, and how you ended up being a game developer and working on A Dark Room?
Amir: I guess a lot of people should be able to relate a little bit to the story. I learned how to build software through just building video games. I think my first thing was, I tried to build a chess board with The Knights Tour. I think I was a sophomore in high school at that point in time, and I had Turbo C++ and I was like, I want to try to build chess.
Pro tip: when you’re just starting to get into game development or any kind of development, chess is probably one of the harder problems you can pick up.
I got my first computer when I was 10 years old. It was a 386 with a turbo button. I played games on there like Scorched Earth. That was one of my first games that I remember. And then I had Chessmaster 3000. And that’s what really got me into it - just being around video games.
And then I just did my thing in high school. I found that I really liked programming, and got a degree at University of Texas at Dallas doing computer science and software engineering. And during that time period, I just finished my degree, and then went into the corporate world. Beause that’s what you’re supposed to do, right? You have all these dreams and aspirations of, “I’m going to build video games,” and all that stuff, and then reality hits, and it’s like, “Well I actually have to make money”.
I became a software developer at that point. I did that for 8 years, and then I went on a sabbatical, and saved up enough money to take some time off. I downsized from a two-bedroom to a one-bedroom. My wife was okay with it. Our standard of living really didn’t change, because I made sure to have a really solid foundation.
I took some time off, and A Dark Room was like - hey, I want to do iOS development, and learn the platform, and this is a really easy, minimalist text-based RPG that I can port over. It wasn’t, but it starts out really simple, and then it becomes really intricate. I ported it, and I was really happy with it. I put it in the App Store, and then I became obsessed with the marketing aspect of it, and building a user base – watching my baby grow.
It went viral, and I guess that’s all she wrote, right?
After that I was able to get a new physician. On the sheet they make you fill out, it says “Occupation”. I was able to put down “Game Developer”. That was a pinnacle moment for me, that I could stop calling myself like a “Software Engineer” or “Software Consultant” and call myself a game developer.
It’s so fun, because you’ll be part of conversations - before it was like, “So what do you do?” And I would say, “I’m a software developer, software engineer”. And then that’s where it stopped, right? Because they’re like, “What does that mean?” My wife would do the same thing. Like someone asks her, “Oh, what is your husband doing?” She goes, “I don’t know. I think he does something with computers”. You always get, “Hey, I’ve got this computer problem. My internet’s not working” or something. Now when I talk with people, it’s like, “Oh yeah, I’m a game developer”, and it completely breaks the ice, and you can start talking about really cool things from that point.
Len: That’s a fantastic story, and I’ve got to say, I can actually sympathize. My first corporate job was working for a company that’s now called Dealogic, in London. I was doing research on mergers and acquisitions basically. But the company at the time was called “Computasoft”. All I had to do was say that name, and people glazed over, not interested.
Amir: Yeah, they’re like, “We don’t care.”
Len: It was just amazing. You don’t even need the full word “computer” in it.
Amir: And it’s so interesting, you start saying “I do software development or programming” [and they reply] “So you’re in IT, I hear that’s not a great industry to be in.” Suddenly they know everything about IT, and they can just group it into this one, umbrella discipline.
Len: I don’t know if this is a tangent, but it’s actually a very interesting topic, and very important, I think. For example, there was a recent exchange I saw on Twitter where a relatively prominent journalist based in New York tweeted about - I think the guy’s the co-founder of Vox, and he had written a Medium post about digital publishing, basically. And then someone sent out a tweet saying like, “How can this guy, a tech administrator, talk about digital media?” To which of course he sort had a really snarky reply. Marc Andreessen retweeted it, saying, “What’s a tech administrator?” ironically. I replied to that with an image from Peter Thiel’s book of the Solyndra guy next to Elon Musk.
Amir: I’ve got to see this post.
Len: There’s a weird preoccupation that people who are unfamiliar with software have, where it’s kind of scary to them, because the person who knows how the computers go, actually runs everything, and knows how everything works. It’s very unsettling to people aren’t familiar with software, but have a prominent position from a kind of legacy, institutional structure.
So they take the opportunity to sort of goad the new media guy, for actually knowing how the computers go, as though that’s somehow evidence that he doesn’t understand the real work of publishing.
Amir: I can empathize with that too. It’s like, when my car breaks down, I take it to the mechanic, and he’s like, “Oh you’ve got this and this wrong.” And you feel so disenfranchised, right? You know that this person has the upper hand. You’ve got to trust them and have that kind of rapport. So it’s frustrating.
I hope in the future more people are more literate at manipulating a general computing machine, and doing something with that. I guess there will come a time.
By extension, I think game development does a really good job of that. Beause when I released ADR to iOS, I had teenagers email me saying, “I want to get into game development, this is one of the first games where I really enjoyed it, and I saw that you don’t have to have crazy graphics, or 3D models or anything to actually build something that people will enjoy playing.” And they could visualize; it was like, “I can create a button. I can put some text on a screen.” These aren’t difficult things, I mean they’re familiar to some extent. And that’s what kind of got them into development.
I had people actually email me, “Here’s this like Python console-based game that I built.” And it’s so awesome. I play, and it’s like, “Oh this is the coolest thing in the world.” To be able to inspire people from that perspective has been pretty fun.
Len: It’s got to be really rewarding to have a community built around it, and to have people emailing you with their creations. And to be able to be a game developer yourself and create things.
You brought up your sabbatical, and I found that really interesting. I was wondering – have your employers since then cared about you taking a sabbatical? Was it something that they asked questions about? Or is it like – they’re just not relevant. You’re a good developer, and that’s it?
Amir: For me, the sabbatical was actually a learning sabbatical. I was primarily a .NET C# developer. And during the, I would say 14, 15 months off, I ramped up on a whole lot of tech, so became really proficient with iOS. I did a lot of Ruby development, primarily for build automation. So I got into Ruby.
I say I’m a Ruby-ist, because I got into Ruby using Rake to build .NET projects. Usually you hear Ruby developers say, “I did Rails.” But I have very little Rails experience. All my Ruby experience is through building iOS games, and doing build automation for .NET.
I finally went into my OSX partition. I had a Mac, but I always used Bootcamp, and I loved the device, and the system itself, I didn’t ever use the Mac partition. So this was an opportunity for me to do that.
When I talk with employers, I say, “Well I took a two year sabbatical to really ramp up on other technologies,” and sell myself as an integration specialist. So not only do I know .NET, I know Rails, I know Node. I can do QA stuff, I can do mobile development. I can speak a lot of different languages with regards to - real languages - interacting with humans.
The fact that I created a product, it makes me a little bit more able to communicate with the business end, saying, “I understand that you’ve got these limitations, or you’ve got these things that you’re trying to accomplish. Let’s see if I can figure something out for you.” So it’s worked out pretty well for me.
The other side is that I’ve kind of become unemployable from that perspective too. Because you start valuing your time so much more. You realize that 9 to 5 on a salary leaves very little for the person, from some perspective. Especially with the market as it is today. I think taking the plunge, and taking the risk, maybe doing independent work, having some kind of side hustle is so valuable.
Some employers are like, “What do you mean you don’t want to go salary?” Or, “What do you mean you don’t want to work 40 hours a week?” It’s just kind of weird for them hearing that from a potential employee.
I do contract work part time. I’ll do six months of contract work, and then when I get the itch to actually do game development, I’ll go back to game development. And go back to contract work and flip-flop. If I find a really interesting project, I might do a year contract. Or if I want to do game development - right now I’m in the game development phase, I’ve laid out my next eight months, and that’s what I’m going to do. And then hopefully, go back to “contracting” afterwards.
Len: That’s a really great story, and it sounds like a pretty cool position to have built for yourself.
I was wondering - I’ve been interviewing Leanpub authors for a while now, and they’re mostly developers or involved with software one way or another. One thing I’ve noticed is that about half studied computer science. I think you graduated about 10 years ago?
Amir: Yeah, December of 2006.
Len: Knowing what you know now, if you were talking to you about to graduate from high school, would you recommend studying computer science?
Amir: I think personally I needed that structure. I loved programming from the get-go. Around 2001, the emphasis on having a computer science degree was much, much heavier. Especially if you didn’t have a public portfolio. GitHub didn’t exist. You couldn’t publish something to an App Store.
I think in this day and age, if someone was like, “Oh I want to get into computer science and programming,” The first thing I would ask them is, “Well do you have an app?” Or, “Show me your website,” or– ‘Cause it’s so much more accessible these days. So I think if I was to graduate high school now, my answer would have been, “Build something, get something out there and show me what you made.”
It’s weird, because you have some more formal professions, like nursing or accounting or business, that I think you really need to have really strong fundamentals that are taught in college. At least for the kind of software that I build, you can hack a lot of things together and get away with it. If I was really interested maybe in like compiler design or distributed systems, or network protocols, I don’t think I could get away without having a formalized education. But at least for the business apps that I build, the crowd apps, product pages, games - you could probably get away with just building something, and then just being hungry from that perspective.
Len: One person that I interviewed recently who’s in software testing said that he would still take four years to study and build things, but he wouldn’t do it at university. It’s this concept of taking a chunk of time at the beginning of your career that you devote specifically to becoming something that you’re not yet.
Amir: Yeah, and recording that.
Len: It’s still an important phase to go through.
Amir: I think another important thing is to just document that in some way, to show to people. So when they say, “Oh you don’t have a degree,” it’s like, “Well, I have 10 apps in the App Store.” Or, “I built these websites.” Or, “I bootstrapped this company,” or, “I helped this mom and pop shop set up a checkout center.” And all those things perk up ears, especially from a junior dev, when you hear that kind of stuff.
Len: Turning to A Dark Room, or ADR as you call it. There’s so much to talk about. It’s been written about on the Huffington Post and the New Yorker and places like that. And you’ve got this great post on Reddit as well, where you talk about what you did, in addition to your book of course, which we’ll get to in a bit.
But I wanted to ask you, there’s a really interesting feature of your success with the game, which is that there’s no reason you can attribute to its taking off. As you said at the beginning, it just went viral. It’s interesting, because playing A Dark Room, unexplained things just suddenly happen, and change the game for you, right?
I was just wondering if you wouldn’t mind talking a little bit about what’s that experience like? When you work on something, it’s been out there for a while, and then inexplicably, it’s suddenly a hit. I mean, what’s that like?
Amir: So the short version is, you’re not prepared for it. I don’t think anything I’ve done in my life could’ve prepared me for getting 20,000 downloads a day for effectively a little bit over two weeks. You wake up in the morning, and you see, because you can go to iTunes Connect and see how many downloads - and you’re at the number one spot, you’re like, “Oh, 20 000 downloads.” It’s like - I’m going to get a 6 figure check from Apple this month. And it’s just surreal.
I lost sleep when it went viral at the end of March in the UK. I have some ideas about how some of the virility occurred. It went viral in the UK at the end of March, and then it took 12 days for it to go viral in the US, and then it stayed there for 18 days. Then it finally fell from the number 1 spot. I did a release, and then it went to the number one spot again for two more days. During that time period, I didn’t sleep. It’s really difficult to put into words.
Len: Can you maybe explain a little bit about, like, how did your day change? Were you getting inundated with messages from people and requests from the media?
Amir: The requests from the media came when I hit the number one spot. That’s when Huffington Post and New Yorker and those people started contacting me. And of course, you say yes to everything, because you’re like yes I’m going to try to push this strain as long as I can. And then you wait.
For me - I actually wrote it in the book as an analogy - it’s like you buy a lottery ticket, and you’re watching the TV, and they give you five of the six numbers. And all five of the numbers are correct. So you might make some money, right? You’re like, “Alright, I’ve got this much in the bank.”
But that sixth number is where your life changes. It’s where you don’t have to worry about that kind of stuff anymore. And the TV cuts off on the sixth number, and you have to wait for some other media outlet to tell you what that sixth number was.
For me, during that time period - I’d do the math, and I’d say, “Okay, this app needs to stay at the number one spot for 62 days, and I never have to worry about money again.” Obviously that didn’t happen, but for that 18 day period, that’s what I was mentally going through. To stay grounded and realistic and say, “This is going to end,” or “This may not happen,” it takes mental effort to not get away with yourself.
Every moment, like every 3 minutes I was checking the app store, “Am I still number one? Am I still number one? Am I still number one?” That didn’t go away during that time period.
At the number one spot, I got about 300 reviews a day. Just reading the reviews, it was nice. It gave me something to do, while I’m toiling away going, “Oh gosh, what’s going to happen?” You can’t release a new version of your game, because you don’t want to lose your reviews, and you don’t want to rock the boat. So I’m in this limbo going, “Okay, I can’t release a new version. I can’t make any changes. There’s bugs that have been found, and I can’t do anything about that. So I’m going to just read reviews and refresh the App Store page, because I have nothing better to do with my time right now.” It’s like that kind of feeling.
Len: You refer to negative reviews in your Reddit post. I was wondering, just thinking about it generally that nowadays with social media and stuff, when you become well known for any reason, it seems to me everyone just suddenly encounters negativity. It’s just a feature of the experience. I was wondering if that happened to you. Did you encounter people saying, “This is a fraud,” or, “This is BS,” or anything obviously, totally, false?
Len: What was your experience like with that, and did you do anything specific to deal with it?
Amir: That was interesting. Negativity is like death and taxes, you’re going to get it. Especially when you’re in the limelight, and you’ve got that big target on your back.
So what happened in the UK – The UK market is, I would say a fifth of the US market. Hitting the number one spot in the US took 20,000 downloads. Hitting the number one spot in the UK took about 5,000. Coincidentally, getting above the fold, like the top 10 games, is significantly less also, right? In the US, it probably takes about 4,000 to 5,000 downloads to get to the number 10 spot in games. In the UK, it was only like 800 to 900 downloads to get there.
What happened was, I got to the 800 to 900 downloads, and I was above the fold. That’s what slingshotted me up to the number one spot in the UK. Up until that time period, my target category was RPG. So the reviews were generally good, because people wanted an RPG, and they downloaded an RPG and that’s what they expected.
But suddenly, I’m in the top 10 games with real games. You’ve got Minecraft just above me, and I’m like, “What’s going on here?” And people - it changes. Suddenly someone downloads the game that’s looking for, I don’t know, like just a regular arcade game. And they’re like, “What is this crap, how did this get here?” So they left one star reviews.
The interesting thing with the review system is the way the reviews are sorted, it’s by the number of stars and then by length. But you hit some threshold where the length becomes more of a factor in the review. I got a few people leave like these really long, one star rants, and they became the top reviews on ADR’s page. It created this herd mentality. Suddenly people that would download this game that had this negative connotation to the game already going into it. They’d play it for a minute and go, “This is stupid. Everyone’s right.” And then they would leave their one star rants.
It snowballed. It got to the point where about 33% of my reviews were 1 star reviews. Initially that was really rough. I would go on Twitter and search for people talking about A Dark Room and try to directly address their issue, saying, “Look, this is a real game I promise. Look at all the things I’ve done, look at my blog. This is not a fake scam game.”
But I think, in retrospect, you can get used to it. Even on the Reddit posts posts that I did, there were comments like, “Oh this guy’s just doing marketing for his own game,” even though there was like a lot of good content. Then there was another one, where it’s like, “Why are you shilling your game out? You’ve already made $800,000 or $700,000. Why do you have to shill your game?” You’re lose either way, right? Because if I made my stuff anonymous and said, “I’m not going to tell you the name of my game,” then I would have gotten negative comments saying that, “I don’t believe you, you’re just blowing smoke up our skirts with regards to what you’ve done.”
So you lose either way. I think with regards to coping, you just get kind of used to it. After the first drove of negative reviews, I found that I was still getting positive reviews. Tou just get used to it, and it rolls off your back after a while.
Len: That’s a really interesting story. You say that you even got negativity about posting these long, really helpful, detailed pieces of advice for people to succeed in the App Store. Even then you get negativity. But what also happened is, I saw someone had tweeted about it. And then, well I mentioned him already, but Marc Andreessen retweeted it. I haven’t had experience with success in the app store or anything like that, but one thing I’ve remarked is that, if you succeed genuinely, smart people, all the smart people, see that.
Len: And so the negativity is coming from - no reason to say the negative term, right? But it’s coming from not those people.
Len: That’s one way at least I’ve found of dealing with it. It’s like there are some people who are just not reachable. That’s just what that person is. If they’ve got a problem, the reason they’re negative, and in the way they’re negative in the first place, is because they’ve got some kind of underlying problem. Thinking about it that way, it’s like, “I can’t solve their problems for them, right? This isn’t my problem. This is their problem.” Anyway, I find that that sort of helps.
Amir: Another thing that I think helped quite a bit for me was, I try to practice empathy. It’s funny, because I was like 28 when I started the sabbatical. And I turned 30, and I don’t know, something clicks in your head. You suddenly stay young, but become more mature from that perspective. So I think back to [when I was] 27, 28. I was like, “Man, I was an idiot and asshole back then.” And then you grow older, and you’re like, “I can do better at it just being more empathetic.” So you see some of the negative reviews, and you’ve got to just think to yourself, “Man, they might be having a bad day, or they might have tried doing the same thing I did and really, really struggled with it.” They just have some resentment that came from their own struggles. So there’s no telling what they could be going through in their life. And I think that really helped also, to ease that.
Len: That’s interesting; that’s another sort of conclusion I’ve come to about dealing with people through a product. It’s that you don’t know what their day’s been like. You don’t know if they’re at a bad point in their life. You really don’t know anything about them. And so someone who’s relating to you negatively might - yeah, you become a lot more empathetic.
One thing I’ve also found too is that treating everybody uniformly in a friendly manner - if there’s a negative interaction, and you come back with that empathy, almost every time you actually get an apology. I think that might be a little bit different on Twitter, because Twitter’s so short form.
Amir: Yeah, the brevity, it just leads to–
Len: But definitely dealing with people over email - in my experience in a number of projects - maybe they just skipped lunch?
And then sort of again, there’s sort of another way of coping with it, where their issue is their issue, not mine.
On the other side of things, what about the positivity? You must have been elated, and it must have been different having people contact you with like, “Hey, here’s my game.” What was that like? Did you have any kind of impostor syndrome thing happening?
Amir: I’ve struggled with impostor syndrome. Even with the new games that I build, I’m like - I can’t top A Dark Room. I’ve got the prequel out there. For those that have read - Flatland by Edwin Abbott (it’s on the Gutenberg Project) - if you haven’t read it, it’s about an 80 page book, and it’s about a two dimensional shape that lives in a two dimensional world. It’s actually a satire around the Age of Versailles, the Versailles time period. It’s told from the perspective of a mathematician, about how this civilization interacts with each other, and how women are treated, and how different caste systems are created, within this two dimensional world.
I built a game around that, that’s my third game I’m building. And I’ve tried desperately to say, I need to get out of the shadow of A Dark Room because I need to prove myself that this wasn’t just a one-shot wonder, or one-hit wonder. So I struggle with that impostor syndrome all the time.
But man, the emails - a lot of that stuff - it is the most rewarding thing I’ve ever experienced. I’ve done a lot of open source, especially during my corporate time. You get occasional thank you’s and, “We really appreciate what you’re doing,” and stuff. But I’ve gotten emails with people saying, “I loved your game, I want to get into game development.” I actually got reviews in the App Store, there was one review on - and the reason I mention A Noble Circle was because it was on that review - the review said, “Today was my 17th birthday, and I don’t think I would have made it past today if it wasn’t for this game.”
How do you internalize that? He was on the verge of saying, “I was going to end my life, and this game kept me from doing that.” When you see reviews like that, it just really puts things into perspective, that you could potentially be changing someone’s life.
Another thing with A Dark Room is that it’s actually playable via voiceover. So people that are blind can actually play the game. It hit the number one spot, and I got an email from a brother who was saying like, “Hey man, I really appreciate it, that you made the game accessible to the blind. Because my sister now, at school, can actually talk about the game with her friends. And she doesn’t feel left out around that kind of stuff.”
So those kind of things - I hate talking about it, I feel like I’m bragging about it, but those kind of things are just immensely rewarding. You can’t put a price tag on that kind of stuff. It’s been humbling, really humbling from that perspective. From what you were saying, with being genuine - I try my best, absolute best to be genuine with all my interactions and whatnot. Hopefully that shows through whenever I write or talk; even in my games, I hope that shows through.
Len: I certainly think it does.
I’ve got a particular little hobbyhorse about how, often when someone has had a success, and they’re asked about it, the story ends up being, “Well I was working in the movie store, and then I was directing Reservoir Dogs.” It’s not the best example, because Tarantino actually has explained how that happened in detail. But often people will tell their story about how they became a success, and you’ll think back on it and be like, “Hey wait a minute, there’s this huge gap.”
But what you’ve done is you’ve filled in that gap. There’s a really humorous moment at the beginning of your Leanpub book, where you talk about how there’s so many questions to ask. Like, “What should my stack be? What should blah, blah, blah, when I’m getting into developing an iOS app?” And then you go, “Well there are lots of things to choose from, but here’s one answer.” And you just list a place to start. The joke is that it ought to be so obvious that people should do this, but most people don’t.
When you’re desperate and you’re starting out, you’re like, just tell me what step one and two are. Then we can move on.
But if I have to know what I’m doing before I start, then I’ll never start. Anyway, I noticed, between your book and Reddit posts and stuff like that, you’re obviously trying to give people that. You’re giving people that equipment right away, which is very generous and certainly something no one has to do.
Len: You’ve mentioned A Noble Circle a couple of times, one of your games. I noticed that you’ve written about that too. You were publishing it while it was in progress, right? So the first version was kind of, as you say, barely playable.
Len: I know that there are in-progress games on Steam, and it’s interesting, because at Leanpub we’re into in-progress book publishing. Even Kanye West is getting in the game. He was in-progress publishing a song.
I’ve never asked someone who’s done this with a game before what that process is like. Do you get a lot of feedback from people? Do you have a plan for your releases? Where like it’s like you set up a surprise release in advance? What’s it like in-progress publishing a game?
Amir: I guess I just realized this idea of incremental development - and this kinda goes back to my corporate experience, because I don’t want to say that my corporate experience wasn’t helpful to me - during my corporate work, I got all the like Agile, Big A Agile stuff and everything, and it helped, because I understood what it meant to actually incrementally build something.
There’s so much pressure for game developers to have this phenomenally perfect release. But we have these facilities to allow us to release incrementally, so why aren’t we using them? That was kind of the idea that I had with A Noble Circle.
So my first version was barely playable, and I made it free. I was like, “Here, this game’s barely playable. I’m going to put it out there.” Then I put a bit of developer commentary at the end, saying that, “Look, the App Store’s broken, or I think it’s broken. We have a lot of ads, we have a lot of in-app purchases, and everyone’s gone for this cash grab without really taking into consideration that you may not need to do that. So what I’m going to do is, I’m going to release a game, and I promise to update it at least monthly. You’ll see these updates come through, and if you get it now, you get it for free. Eventually it’s going to cost money, but right now I don’t feel it’s worth money. So just take this journey with me. Not only are you going to get a completed product, but you’re going to see the story of how I got there.”
And that’s been really great. You get negative feedback, something like, “The jump mechanic’s stupid,” or things like that. I listen to that and I incorporate it. But then you have people that say, “Yes! There’s another update, I can’t wait to try it out, and see what he did differently, or what changes did he make.” They love that aspect of it. because they replay the game and go, “Oh yeah, he added this new thing.” Or, “I like the changes that he made. They’re already adding a new chapter.”
It’s wonderful. It’s still not done. But I’m charging a dollar for it now. I released a free version, A Noble Circle - Prologue, that had that a much more polished version of the initial experience. There’s an interstitial to say, “Here’s what I think is wrong. Here’s what I’m trying differently. Go buy my other game.” Or, “Go buy the full version if you want to continue the story.”
I hear game developers say, “I’m going to start a Kickstarter.” But I quasi did that myself. I just did my Kickstarter in the App Store, and said, “If you like what I did here, buy the full version, buy A Dark Room buy The Ensign, you have different options. And if you didn’t, no problem. I mean, you didn’t lose anything, hopefully you played a little bit of it, and maybe you can give me some feedback on that.” But generally speaking, all the reviews have been very positive. This approach has been really exciting for the end user, and for me too, because I just enjoy doing it.
Len: Can you get refunds in the App Store?
Amir: You can, but it’s tricky. They don’t advertise it, but on the receipt that you get, it’ll say, “You’ve purchased this game”, and there’s two links. You can gift the game to someone, and you can report a problem with the app. If you click the “report a problem” button, you can actually get a refund, and there’s a dropdown list of why you wanted to get a refund. Things like, “I accidentally bought it,” or, “It wasn’t what I expected it to be.” Generally speaking they do give the refund. But they don’t advertise it. They say, “All sales are final. No refunds.” But you can get a refund if you go through that process.
Len: That’s really interesting, because as you pointed out, there’s all these tools to publish incrementally. And that can be various types of products. But it’s especially, obviously, suited to digital products. I have this view that we’re working out what commerce is like around things that aren’t finished.
Amir: Yeah, and that’s why I like Leanpub too, from that perspective. I can put something out there that I’ve written. How it started, was I had the blog entry. And I had all this blog information in there. The first step was, take that blog, put it in the book, and then write a postmortem on all the postmortems that I wrote. That worked out really well, because I could reflect back saying, “During this time period, I thought that this was the problem. But this was really what was happening during that time period.”
Len: I actually have a couple of questions about that that I’ll ask in a couple of minutes. The theory that I have is that, especially if a digital good is published in-progress, but also self-published, where there isn’t what you could call a triple-A company behind it, then putting forward the fact that it’s easy to get a refund is a very important step in establishing a positive relationship with a person who’s buying the thing, and the person who’s making the thing.
Because then, there’s this positivity and this openness to it right from the start. It’s like, “We trust you.” Your first encounter with us, is us trusting you and asking you to place some trust in us. It’s like, “you’ve released something early and unfinished, but I’m trusting you that you’re going to do what you say you’re going to do. My theory is that surfacing the ability to easily get a refund for a product you can download actually drives sales, especially in the long run.
Amir: You can see the other side of that coin. I try not to worry about it, but there was an indie game development company that built a game called, “Firewatcher.” It’s on Steam right now. If you haven’t played Firewatcher, you can get it on PS4, you can get it on Steam. I don’t know if it’s on XBOX One. They call it a walking simulator. You’re basically this guy that’s at Yellowstone National Park, and you’re making sure that there’s no forest fires. But there’s this whole story and this relationship that you build with another person that’s also watching a fire in a different quadrant of Yellowstone, and uncovering your own past, what happened to you and why you’re there, what happened. There’s no way to die. There are no objectives you can fail at. It’s basically a walking simulator, where you pick up things and talk on a walkie-talkie with this other person.
What happened on Steam was that it was selling for like 15 bucks. And this person was like, “Yeah, I got this game. I beat it in about three hours, and I want a refund, because I could do something else with those 15 bucks. I mean, I loved the game but what should I do?” The developer actually responded to that entry. At the end, the person said, I’m not going to get the refund. But it’s really weird that even in this digital age, we still haven’t found a way to assign value to something that you can’t touch and feel.
That’s something that that everyone in the App Store struggles with. Especially premium game makers. It’s so difficult. I can buy a $700 device, but I can’t sell a game for 99 cents, without people feeling like they’ve been cheated if they don’t like it. I’m hoping as time goes on and we get more digital things, people just are okay with it. But that’s been a struggle that I’ve seen personally myself.
Len: It’s in that broader class of the way we’re in engaging new forms of commerce now, that have been enabled by certain types of technology. There’s no common sense yet. It’s so new we haven’t had time to develop. Remember how people used to leave their ringers on their mobile phones on, when they were walking around? You basically never hear a phone go off anymore.
Amir: You don’t hear the custom ringtones that they had?
Len: When they first came out, people would do that. It takes a while for codes and conventions to be built.
One thing we’ve definitely experienced at Leanpub is, we surface to people that refunds are two clicks away. That means that yes, if you want to, you can buy a book, and download it, and then you can get your refund. I mean obviously we don’t encourage that. But it’s rare.
Amir: Yeah, and you try not to sweat that either.
Len: Well you can’t, I mean with BitTorrent and whatever. This is my personal opinion. People in this industry obviously have many different opinions about it. As we’re talking about it, people are still working it out.
My view is that, especially if you’re a creator, if you’re creating things, you should spend your time creating things. Even if it were possible to kind of diminish the pirating of your work, you’re not the right person to be doing that. You should be writing your book. You should be thinking about your novel. You should be thinking about your game. You should be working on your song or whatever it is, and just really genuinely leave that out of your head space, and concentrate on being better at what you’re doing.
Amir: For me it goes back to the empathy aspect, too. If someone downloads my book or pirates it, it’s like, “Maybe they can’t afford it.” Even with my game, I had people on Reddit, when I would give away promo codes, say “Hey, this promo code doesn’t work in South Africa. I can’t use it here. And I really can’t buy your game either. Because there’s no really good means for me to buy something. Because I don’t have a bank account or credit card.” Or it’d be a kid: “I’m 13 years old. I get an allowance, I can’t do this.” So sometimes I just give them the benefit of the doubt: there’s a reason why they can’t shell over the cash.
Len: I couldn’t agree more. It’s interesting - the best-selling Leanpub book in 2015, in terms of copies and revenue - which are two separate calculations - had a minimum price of zero. The book that made the most money, you could get for free.
I like to say, this isn’t us being precious. It’s a serious business model: giving that flexibility to people is actually beneficial. Definitely, my personal belief is that spending time and energy in attacking people is really bad. That is counterproductive activity.
Amir: It’s just not worth it.
Len: It’s interesting you’re saying about how they kind of hide refunds at the App Store. It used to be, when I was a kid, if you like got your remote controlled car for Christmas or your birthday, and it didn’t work, and you took it back, you had to engage in a negotiation, and prove that you didn’t do it. You stand in line, and they treat you in a sinister fashion, like maybe you were trying to hoodwink them or something. The last time I returned something was at a Best Buy or something, and they’re just like, “Okay, got the receipt?” And then I handed it over, and they gave me my money back. They saved themselves so much money, and made a happy customer, by not being jerks about the exchange of money and goods.
Hopefully we’ll get to the point where it’s just common sense that being negative to everybody - like DRM on an ebook for example - that treating all of your potential customers like you think they might be bad actors is just a terrible approach. The people who are kind are going to win the hard ass war.
Just moving on actually, I wanted to ask you a couple of questions about your book.
Surviving the App Store is a great title by the way. You did a really interesting thing where you had this very early release, where you had a promotion, where you can get it for a big discount if you buy it now, but there isn’t any content in it yet. I was wondering how that worked out?
Amir: It worked. It was the craziest thing. I did the free thing. I had just one page, and it was like, “This is kinda what my table of contents would be.” I had a whole “Hello World”. I set the price to free, and $4.99. I tweeted about it, I had the cover. I did my cover in Photoshop, and I’m terrible at it. You can actually see the progression, I’ve got some tweets with the old cover on there. I’ve got some feedback on that too, which was cool.
But I put it out there, and I was like, “Somebody talk me out of this please.” Somebody talk me out of writing a book. And then right off the bat, I got like insta buys. I got like five or six people paying me the five bucks for the book. I’m like, “You guys are crazy. I don’t have any pages, and you’re buying my book. What’s wrong with you?” And then from there, I was like, “Okay well, let me take my blog. My blog is still up, so you can read at least the initial response that I had. It’s like a 14 month diary of everything that I went through. So you can still read that.” And I was like, “Okay, let me copy and paste that in there, in the book. And then read through it and then have a postmortem on my postmortems.” That was the first exclusive content that I added to the book.
I kept it free during that time period, and just said, “Here’s the update to the book.” I kept on getting good feedback and good sales from it. It’s surreal that I can sell a promise - that I built up at least some credibility that says, “I promise to do this. I have no pages. Give me $5.” And people did it, and it was just awesome. It’s a really great feeling.
Len: I was wondering actually, I mean the answer might be in there, but why did you choose to use Leanpub rather than any of the other platforms you could’ve used?
Amir: I think that was the one that I heard the most about. The GitHub integration was my primary point - that I could just have Markdown files and have the GitHub stuff. It just worked for me, so that’s the model that I went with.
Len: Okay so the Markdown was a selling point?
Amir: Yeah, Markdown and the GitHub integration was the big selling point for me. And a part of me was like, “Oh I can make the repo public”, but I decided to keep it private, during my editing phase and whatnot, and they can get the book. If you really tried hard, you could probably go through all my Reddit entries and glean some aspects of it, but at that point it’s like, email me and I’ll give you a free copy of the book, just do that.
Another interesting angle that I had with the book was - and I’m still playing around with it, especially if I get some more traction - is to keep it free. Right now it’s $4.99, the minimum price. But I might make it free again. Because it’s free publicity for my games, and that’s where my income really comes from, is the games themselves.
Is there a service that I can set up, where if you email me your iTunes receipt, and I see that you’ve bought my game, that I could give you the book for free? Or some variation of that? There’s no public API to get sales information from Apple. Those are the weird things that I’m thinking about right now. How can I drive more sales to my game? And then you get a lot of the - the 90%. The other 90% for free. So just letting my wheels spin on that. I think it’s doable. I think it is actually really doable. But I just have to see if I can figure it out.
Len: That you could do it the other way around, where if someone bought your book, you could give them the game for free?
Amir: I get 100 promotion codes per release. So I could do that 100 times, but that’s where it stops, right? I was thinking, I could just start up a daemon process and send AWS instance where you email this daemon process, and then it passes the receipt. And then that gets you a coupon code, gives you the coupon reference, so that you can download the book for free.
And then you start thinking about, well they might be able to forge the receipt. And it’s like you can’t start thinking about that kind of stuff, you just have to concentrate on those things.
I think I emailed you before about this - a means to say, have you subscribed to my newsletter, and get a discount on the book, and just facilitate that through Leanpub - that’d be great. There are different ideas there too. I use MailChimp right now, and trying to figure out ways to integrate potentially - if you subscribe, then send off this subsequent thing to give you the coupon code for the book. Yeah, different ideas there too.
Len: My last question related to that is - the theme of your book and your posts is dealing with Apple, as an iOS product maker. I was wondering if you could just, for a couple of minutes, talk about what’s that experience like?
Amir: It’s definitely, don’t call us, we’ll call you. There’s two sides of it. The editorial team is this untouchable group. Developers don’t have direct access to them. So I have a liaison to the editorial team that will advocate for me. The good thing about these liaisons, they’re human beings. You can talk to them and send out a quick email saying, “Hey man, I’m doing this.” You don’t have to worry about being really formal with them and everything.
But it is basically a black hole. You send things over, and you get very little feedback coming back on the other end, outside of the Thursday during lunchtime in Cupertion where you get to see if anything really happened. But I guess, they’re in a position where they can do that.
Len: When you say editorial, is that both approving the app or a new release, and positioning in the App Store?
Amir: This is strictly for featuring. Curation of featured content. As far as the ranking system, it’s pretty much completely calculated off of the number of downloads you have. So from what I’ve seen, it’s about a four to five [day] moving average of your downloads. So getting 600 to 700 downloads a day on the premium App Store, will put you in the top 150 games. Getting 6,000 to 7,000 downloads a day in the free App Store, will put you in the top 150. You could calculate it very easily, if you have the entire range – which I do!
But yeah, the entire feature process is 100% curated. There’s no automatic engine. There are people that look at apps, interact with developers. Of course interact with Triple A companies. And they figure out what they want to feature, and what they want to push. They try to curate that content. I think that’s where the value is in the App Store. I haven’t done much Google Play yet, I’m going to - I’m porting the game over right now.
But as far as Apple goes, yeah, there’s a lot of work that goes into getting good content out there. And frankly, it’s good content. So when you see something featured, it’s worth buying. I can say that pretty strongly. At least if it’s not a Triple A company, right? If it’s some unknown company, if it’s not SquareSoft, Kim Kardashian, King, Warner Brothers - buy it. You’ll enjoy whatever it is.
Len: That’s really great to know that detail, and that they’re doing a good job over there. Sometimes, as a non-expert consumer, it’s hard to know.
My last question is, if you could ask us to build one feature or solve one problem that you encountered at Leanpub, what would you choose?
Amir: I think it’d be the newsletter thing. If I can collect an email for a discount - huge. I think that’d be pretty big for me.
Len: So what do you mean? Can you say maybe a little bit more about that? Because currently we have a feature where, when someone’s buying a book, they can choose to share their email address with you. And you’re saying you want something more than that.
Amir: Right. If they share their email address, I give them a discount.
Len: Oh I see, yeah got it, understood. Yeah, that’s something we’ve thought about before.
Amir: That’d be really cool from my perspective. Because I’ve got the blog, and the more people that I have on the blog, I can subsequently have different entries around game development. I can talk about my Twitch channel if I want to start that up, screencasts, and of course, cross-promote my games. So that’d be pretty valuable and I do believe I give good content outside of the book.
Len: That’s really interesting actually, on the theme of things that have changed because of technology and the types of products that we can create and sell to people, and that people can buy. There’s a certain category of author who comes to us thinking that by default you get a customer’s email address, which to me is just like, that would never occur to me that, if I’m buying a book that the creator of it is going to get my email address.
Amir: Yeah, that’s personal.
Len: Yeah, exactly. Like I said, we’re working out the codes, right? There are some people like - especially people who live in the newsletter marketing world -who often are like, “That’s what I’m building, my email list.” We’re hearing things about Bernie Sanders’ email list basically is this hugely valuable thing that the Clinton team is trying to get him to hand over and stuff. There are people who think from the email out, right? It’s something we encounter. But definitely giving people the option to share their email address with an author is something we really like. Because we want to encourage people to establish a relationship with each other.
Amir: And out of all the subscriptions, only one person has opted out, right? So everyone’s been really good about giving their email.
Len: On that though, you actually have to opt in.
Amir: Yeah, yeah. Sorry.
Len: That means everybody but one person has actively opted in. So that says a lot!
Thanks a lot for taking your time to talk about this. It was really interesting chat. And thanks for being a Leanpub author too!
Amir: Oh I love it.
published Jun 10, 2016
PayPal has suspended operations in Turkey.
Leanpub uses PayPal both to process payments to authors and to process purchases from customers. We take PayPal and credit cards, but our credit card form is processed by PayPal Website Payments Pro. So if PayPal does not support a country, neither does Leanpub.
The most important effect of this is that authors with Turkish PayPal accounts will have to find other PayPal accounts, that are eligible to receive payments, before we can make royalty payments to them in the future.
We are emailing all authors of Turkish language books directly, and we have posted on our Google Group as well as here. (We are aware that there may be some Turkish authors of non-Turkish-language books, and authors of Turkish language books who are not based in Turkey, but this seems like the appropriate level of email.)
This also means a small loss of sales to all Leanpub authors who have customers in Turkey.
We thank everyone for their patience during this unfortunate situation.
published Jun 10, 2016
Nick Sutterer is a Sydney-based software developer and popular conference speaker, with a particular interest in open source frameworks and software architecture. In this interview, Leanpub co-founders Len Epp and Peter Armstrong talk with Nick about his career, about his book, and about self-publishing on Leanpub.
This interview was recorded on April 26, 2016.
This interview has been edited for conciseness and clarity.
Len: Hi, I’m Len Epp from Leanpub, and in this Leanpub Podcast I’ll be interviewing Nick Sutterer.
Nick is the author of the Leanpub book Trailblazer: A New Architecture For Rails. The book helps you build a complete Rails application using Trailblazer, a thin layer on top of Rails Nick built that brings a high level application architecture, decent encapsulation of the new code structure. In Nick’s own words, “The book takes you for an adventure trip from Rails code jungle, to the Trailblazer beach, and involves building a complete Rails application using all of Trailblazer’s goodies well, exploiting the Rails way where it helps.”
Based in Sydney, Australia and originally from Germany, Nick is a software developer and popular conference speaker, with a particular interest in open source frameworks and software architecture. He blogs nicksda.apotomo.de and you can follow him on Twitter at @apotonick, and you can also find Trailblazer on GitHub at github.com/trailblazer.
In this interview, we’re going to talk about Nick’s professional interests, his work building Trailblazer, his book, and at the end, his experience using Leanpub and ways we can improve it.
Just before I start asking questions, I want to say that we’re joined today by my Leanpub co-founder, Peter Armstrong. He’s going to be asking some questions later on. So, welcome Peter.
Len: I usually like to start these interviews by asking people for their origin story. I was wondering if you could tell me how you got interested in software, and specifically what led you to build Trailblazer?
Nick: It all started with computer games. I was one of the lucky kids who had access to computers really early. So we’re talking about the early 80’s. I was playing black and white computer games, and I really got intrigued by the idea of making my own games. So that’s how I got into programming. Of course, I’ve never managed to write a game. But I started playing around with HyperCard on really old Apples, and moved on to C++, and 15 years later, ended up with Ruby. That was a nice development.
Len: I found a talk online, I think it might have been in India, where you said that you wouldn’t be a programmer if it wasn’t for open source. I was wondering if you could explain why that is?
Nick: Wow, so you actually watched my talks, so that’s great. As I said, I started programming following this idea of writing computer games. I was maybe ten years old, and you are not going to write a computer game when you’re ten years old. So I kept playing with lots of languages, and I think I started Perl, and I got intrigued by this idea of object oriented programming encapsulation - even though I did not have any idea what real encapsulation looks like in an actual program.
I was also reading about TCP/IP Illustrated, and this taught me a lot about layering. So I learnt a lot about like - this layer isn’t supposed to know about the other layer. So, I kind of saw the similarity to object oriented programming, and I started playing with a widget framework written in Perl. Of course, not a single company on this planet is using it. But that was when I had my first feedback from actual users or people trying it.
By that time, I was still working as a Perl/PHP web dev. But I wasn’t really loving working on other people’s products only. So the open source stuff gave me gave me some kind of new direction and motivation to keep working in the web environment. And it’s absolutely true, if I wouldn’t have discovered Ruby, and if I wouldn’t have discovered all the problems in Rails that I was trying to solve, I probably would’ve ended up in a different discipline. Something probably like a game developer working for some awesome company, and working on Assassin’s Creed 500. Who knows if that would’ve been the better choice?
As soon as I got into Ruby, and I had this little bit of experience from that Perl project, I started working on open source. And people started actually using it. That was amazing. It’s still amazing. I got feedback - people telling me, “Hey, I forked your project and looked at it”, and I was like, “Oh, this person’s so much better than I am.” And this is how I kept doing what I do for the last 10 years.
Len: It’s really interesting the experience that one can have, if you get early feedback on things from people using it, and what a motivator that can be.
About Trailblazer. I was wondering if you could explain a little bit about what it is, and why you built it. What problem is it solving?
Nick: So Trailblazer, if you have a superficial look at it, it only adds technical complexity to a framework that is already complex, like Rails or Sinatra or whatever. The goal is to add abstraction layers on top of the existing primitive - primitive not in a bad way, but it is primitive on the MVC stack.
So what Trailblazer basically says is: Writing or architecting complex web applications, and that’s what we all do every day, is a complicated task, and having two or three buckets for all our code - M, V, and C, and we don’t put code into Views so we only have M and C - two buckets for complex apps is not enough.
We need more layers to structure code to know where to put certain requirements, to know where to put implementations, and to make it also a bit more straightforward how to test specific functionality.
So what Trailblazer does is - it says, “Use Rails or use Hanami or use Sinatra or whatever” - as what I call an infrastructure framework. It gives us all the task generators, it gives us routing, it gives us HTTP endpoints, also known as controllers. And on top of that, introduce additional abstractions, like deserialization, validation, callback objects and all this kind of stuff.
The additional layers streamline our everyday task of writing functions for web applications. And then, again, go back to the original stack, and use whatever persistence layer is desired. It could SQL, it could Active Record, it could be a Hanami model or ROM, or whatever. We basically give them abstractions from HTTP till persistence layer.
It was a little bit chaotic in the early days of Trailblazer, because we didn’t really know how to communicate what we are doing. But now that we have this word called “high-level architecture”, if people ask me, “What is Trailblazer?” I say, “High level architecture.” And then they either run away and say, “That’s weird.” Or they say, “What is a high level architecture?” And that’s exactly the same abstraction layers that we provide- here’s a bit more layering, a bit more structuring, more components to actually implement your web request. And that’s what Trailblazer does.
Len: Fantastic. I think Peter probably has a couple of questions he’d like to ask about that.
Peter: Sure, so when I was reading the introduction in the first chapter of your book, I found myself agreeing with a lot of your parts of your descriptions and the problems that exist in Rails. So I was wondering what you’d say the main problems of Rails that Trailblazer addresses are?
Nick: So the thing is, I never wrote Trailblazer or the gems–so Trailblazer’s only a collection of a lot of gems I’ve written in the last ten years–I never wrote them just because I wanted to write gems and feel great about being an open source developer who doesn’t get paid for a job anyway. It was always tackling a specific problem and so - I mean I could go into details of all the gems now, but Trailblazer is solving the problem of, “Where do we put a specific kind of code?” So Trailblazer has a bucket for any kind of implementational code snippet you have. So we have buckets for authorization, authentication, for callbacks decoupled from models and for rendering JSON documents, for de-serializing documents, for validating object graphs.
So we have a handful of buckets to put your code into. That is a problem in Rails I’ve seen in hundreds of apps being solved in different ways. So every big project has a different architecture. And there’s zero conventions beyond how to name table names, and how to name controllers. Because that’s what Rails conventions are. They are really primitive.
We give them a strong, high-level and really opinionated structure for their code and their implementations.
Also Trailblazer is solving the problem of testing in a really different way than Rails. So when I arrived, [looking at] pure Rails applications, I always found myself wondering, where do I test stuff? Is that an action, is it a controller test, is it a view test, is that a model test, is that a unit test, is that an integration test?
In Trailblazer we go a completely different way. We say: Decouple all the business logic into specific objects. We call them operations. So all the business logic from deserializing the incoming data to persisting - whatever you want to do - is happening in an encapsulated object in an operation. So we only have unit tests for operations, and we have integration tests to see if the controller wiring works. It’s always really clear when you write a new feature, what to test, and it’s also way faster, because you don’t have to do everything via HTTP tests.
The funny thing is, that wasn’t really planned. But it turned out to be a great feature that a lot of Trailblazer developers love, that we really have a distinct way of saying, “This is what you have to test, and this is how you test it.” And that’s it. So no thinking, “Oh, what should I do? Should I go write another model test? Or should I write another controller test?”
For example, we don’t have controller tests at all. We have integration tests, and that’s it.
The third thing we solve is, that people learn a bit more about object orientation. Because since we introduced all those layers, and all those layers are basically implemented as Ruby objects, people learn, “Okay, this is how object oriented design was originally intended”.
Rails is absolutely misleading you when it comes to object orientation. Just because we have model objects, or we have the controller object, doesn’t mean that this is object oriented. By introducing smaller objects with a limited scope of jobs, people learn, “Okay this is actually probably closer to what object orientation means”.
I’m not saying that we are doing everything right, and everything is the best and Martin Fowler will read Trailblazer code and will say, “This is exactly what I thought about object orientation”. No. But I think it’s closer to the original idea, and the benefit is that you have a really quick understanding of what the app is doing, because every object is only doing one thing, and has, probably has, only one public method. And there’s no way to screw up internal state.
That’s what lots of people told me, is that they after working with Trailblazer, they started to feel what a good object design looks like. Whereas in Rails, you only know, “I have a model, and my model has 500 methods, and that’s called object-oriented design”.
Peter: Right. So Trailblazer is an architectural style, but it’s also a collection of gems. So for someone who’s brand new to Trailblazer, can you give a high level overview of the different gems, but also, I’m more interested in understanding when you realized that you were creating Trailblazer? I mean, you started with one gem and then moved on. I’m curious about how it evolved in your mind, about what you were doing?
Nick: That’s a nice question. I’ll start chronologically, and it’ll all make sense. Once I started writing Rails [apps], I wanted to have a reusable sidebar view component or something, and I couldn’t find anything like that. They told me, use partials and helpers and controller and blah, blah, blah. So I was basically spreading the logic across the entire stack, and I didn’t feel right. I wanted to have a reusable, encapsulated component I can reuse in every Rails controller action, and have my sidebar rendered.
So I started working on that gem called Cells, which introduces what we call “view models” into Rails. It basically gives you components, and the component, all it does is, it’s an object, and it can render a template, and it can execute arbitrary code. So basically it’s a partial, an encapsulated partial. The partial doesn’t have access to the rest of the stack. And so that was the view gem called Cells. Cells is part of Trailblazer, but it doesn’t have any wiring to Trailblazer. It works without Trailblazer, and that’s good, because it’s the most popular actual view replacement of - actually the only replacement for Rails. So lots of people use Cells in thousands and hundreds of thousands of projects.
That was my first gem, and I realized, if there is interest in this kind of stuff, then it can’t be all too wrong to work on alternative patterns in Rails. So I worked on a lot of other gems, and they were part of Trailblazer, including Representable and ROAR.
They’re basically a document parsing and document rendering library. You can grab a document and parse it into an object graph in Ruby, and you can grab any Ruby object or an object graph - nested objects - and render that into, let’s say, a JSON document. That is another gem I was working on. That’s helpful if you write document APIs, like JSON APIs or XML APIs. I know no one’s writing XML anymore, but people still have lots of questions about how to render and parse XML. But that is, again, something completely decoupled from Trailblazer - helping you with rendering and parsing documents in Ruby.
After that, I started working on Reform, which is basically saying - we don’t want to have validations in our models, because sometimes I have a form that might incorporate two or three models, or something that is not even a model at all. And I still want to have validations, I still want to define my fields. So Reform, basically it says, “I’m a form, tell me what are my fields, tell me what’s my nesting, and tell me what are my validations? I’ll help you do the validations, and I’ll help you write those validated data - hashes, or whatever, back to whatever models you want me to”.
So we took the validation out of the model layer in Rails, and put it into a separate validation layer. And so we had views, we had rendering, parsing of documents, and we had Reform, the form object.
And that’s when I started to realize, hmm. The way I wrote controllers in Rails changed, because I was always using the form object, I was always using a new component. And in document APIs, I would always use the representers from ROAR Representable. So I came up with a concept called “Operation”, which is now the central notion in Trailblazer, which is what wraps your business logic. And so Operation is in the Trailblazer gem.
How did I name this “Operation”? The gem name Operation is already taken. I could just start a new framework, like a meta-framework… Whatever, what do I call it?
I was looking on my desk, and there was an old newspaper, and it said like, “Blah, blah, blah is a trailblazer.” And I’m like, “Hey wait, Trailblazer’s a cool name.” So I named the gem Trailblazer, put in the Operation pattern. Only the Operation gem, like the operation pattern was the only thing in the gem. And so the Operation, Representers connects form objects and connects your persistence layer. It’s just an orchestrating object.
Once this gem was there, and the Operation pattern became more and more clear as wrapping your business logic, and orchestrating to all the other stakeholder objects - new things popped up, like policy objects and callback objects and all this kind of stuff. But the main road was really Cells, and then Reform, and once we had Reform, we needed a place to orchestrate it - how the form interacts with the outside world. And that’s where Operation came from, and that’s how Trailblazer kind of started. Then, on top of that, lots of other things were added.
Peter: Cool. I have one more question, and this is a really selfish one.
Nick: Yeah of course. A lot of people do that actually. In my current project, we do the same. We have React front ends, and we have a Trailblazer back end. The only thing we do not use is the Cells gem, the view component gem, because that is supposed to render HTML. You can even use Cells with React. It kind of helps. What you do is, you leave out the view layer, and you still use the deserialization, validation, the persistance layer - all that stuff that Trailblazer structures in your Rails, or whatever.
What we had, and it’s pretty cool actually is, we had this back end in Rails, and we started to port certain actions or certain functionalities into Trailblazer operations. So the controller would only delegate or despatch to an operation directly, without any logic in the controller, because that’s what Trailblazer wants you to do.
Eventually, we could kind of swap the underlying framework. So we could say, “See you later Rails”, And we started using another framework called grape. So we swapped the underlying framework, because Trailblazer allowed us to decouple all our logic from the actual framework. But we still use Trailblazer in a React backend, basically. That answers your question I hope.
Len: Is there a specific intended audience for your book, Trailblazer? Or is it just for anybody who might be interested in learning the framework?
Nick: I didn’t really know who was supposed to be the audience. When I started writing the book, it turned out that the audience for Trailblazer is, all levels of developers. We’ve had people who’ve been around for 15 years checking it out and using it or not using it. We’ve had people who just started Ruby five days ago, and they were intrigued by the idea of layering. And just because there’s more technical complexity doesn’t mean that it’s more complex. The opposite is the case.
The book tries to propagate that on every one of the 308 pages. We are walking people step-wise to master the technical complexity, as people call it. But technical complexity doesn’t mean it’s harder to understand. The opposite. You have smaller objects, easier and nicely layered implementation components that make it easier to understand how software works, and not harder.
So the audience - and it’s really interesting. If you’re on the Trailblazer GitHub channel, you will have people from all kinds of companies, from all kinds of environments. It’s really interesting. We don’t have like a target audience. We try to target everyone. And that’s working; when I check the book and the readers, there’s all kinds of readers, which is great.
Len: Your book’s been quite successful, and I was wondering if there’s anything special that you did in order to achieve that? Obviously, besides writing a really good book and having an existing community already on GitHub, was there anything else that you did to promote it?
Nick: Just in case you didn’t notice, I have the coolest book cover in the world.
Peter: So who did the art for the cover and the cartoons inside?
Nick: There is an artist, his name is Josh Bourne. He’s American, but he lives in Berlin, anh he’s best friends with one of my best, oldest friends. They’re both artists, and they share the same studio in Berlin. I was originally asking my friend, but he was busy. So he told me to get in touch with Josh, and this was just the best thing ever. We would start with little emails. He would send me scribbles and stuff, and the outcome was we have the best book cover, in my opinion, and lots of cool illustrations in the book, and stickers. His style just fits the whole trailblazer project. So that was a lot of fun.
I don’t think the book cover is actually what makes people buy the book. I think it’s driving more people away because it looks really unprofessional. The thing is, I have a huge community of people using one or two of my gems. Those people were excited when they heard, “Okay there’s some umbrella project starting”. And I think that’s what makes people buy it.
The other thing is, we haven’t really started any marketing campaigns. So I think we could do way better. But I’m pretty sure that Leanpub has a lot of tips and pointers for aspiring authors, how to sell their books in a more successful way.
Len: It’s a bit of a dark art, right? One of the things that we’ve noticed is that already having a community built around the subject that you’re writing about is a really important factor in that.
On that note, one thing I noticed is that on your landing page for your Leanpub book, you have a little note saying, Email me for a 50% off coupon if you’re in a country where the US exchange rate is really expensive. I was wondering, did people ask you for that, or is that something that you just wanted to do on your own?
Nick: Both. I got a lot of emails from people saying, “Hey man, I really want to read your book, but I live in”, I don’t know, Ukraine or in Poland or, not Poland, but Brazil. “And $39 is equal to what I earn in three days”, or “My lunch in a week.” I mean, because I don’t have the currency exchange rates on top of my head, I’m not a hedge fund banker.
So that made me think, I’ve got let’s say two or three people per week asking me, and I don’t have a problem [with this]. I don’t do this for money, I do this so people have happy software architectures, and go home happy. And of course, some dollars is nice. So I put this on the page.
Since I’d put this note on the page, it has increased. I get even more emails from people asking me, “Hey, please - I’m from Vietnam, I can’t afford this. This is like seven lunches. Can I get a coupon?”
So it kind of works, and it’s pretty cool actually, because I can totally see that people in the US don’t mind 39 bucks, or people in Australia, because, I mean, a six-pack of beer in Australia is $20, so - you know what I mean? But if the exchange rate is not that good, you have to help them. I mean, I don’t know about the plans of Leanpub, are there going to be different pricing tiers for different countries, or - ?
Peter: It’s interesting. We’ve kicked that idea around. We try to do things that let authors have as much control as possible, in deciding how their book is sold and presented. For example, we’re based in Canada, but we have to charge VAT for our EU customers. That’s tricky, because we have this minimum and suggested price, but then VAT gets tacked on.
So, if you have a nice happy like $9 or $19 as your book minimum price, then it turns into like $26.40 in the EU or in random EU countries with all their different prices.
So we were thinking, “Hey, should we allow authors to say, ‘My book is this price in the EU.’ Or, ‘Hey, my book is this price in like Western world, but cheaper say in developing countries.’” So it’s interesting for us to see this, because we though, “Hmm, maybe this should be a larger feature.”
The flip side is that one thing we don’t like is the idea that the world should be split up and only books should only be available in certain countries… We like the idea that you publish a book on Leanpub, and it’s available in the whole world. As opposed to lots of publishers who don’t even make their books available in lots of countries. We want to make sure our books are available everywhere. So we’re exploring the idea, is the short version.
Nick: I can totally see how hard it is, because you don’t want to make it a regional, localized… I mean, if you start doing like, “Where are you from?” “Ukraine.” “Okay your price is this and that”, I mean, this makes it a bit odd.
Peter: And people will get upset, because they’re on a VPN, say. So someone’s like, “Why did the price change?” And it’s like they were in an incognito window or a VPN, and they saw one price, because we thought they were in the US. And then they went to the checkout page, and we realized, “Oh you’re in Ukraine”. Or you’re in England, and so we stick on the VAT, because we have to charge it based on two or three criteria.
And then it’s like, we’re not changing the prices, this is just VAT being applied. But if people think that things are moving under their feet, they get really upset sometimes. Sometime in the next few weeks, we’re going to be shipping our new storefront, which will handle this a bit nicer, I think. It’ll avoid some of these problems around VAT when it gets shown. But yeah, basically people want to think that the price is fair, regardless of where they are…
Len: It’s really interesting for me as well, because one of the things that’s very important to us, is establishing a connection between an author and a reader, and choosing how much to pay is one of the cool ways that that starts to happen on Leanpub. So it made me happy when I saw your little note. Like, “Email me for a coupon for half off if it’s too expensive”. It was just great.
On that note, in your book, you suggest to people that they’re free to email you any time. And you said earlier in this interview that when you got interested in programming, partly it was feedback from people and getting things out there that helped encourage you. I was wondering if you’ve been emailed by readers asking you questions about the book, or asking you to add things, or even typos, or anything like that?
Nick: Oh yeah, in the early days, especially before I published the book. What was interesting was - and that’s one of the many reasons I love Leanpub - I could push out my book without being finished, and people would start reading it. I had more than 500 readers before the book got published, and that was great. It was a different thinking - now it’s published, now I have less feedback via email, I have more feedback on the official chat channel.
Before it was published, people would really email me. I had so many people telling me, “Hey, on this page you have this and that syntax error.” Or, “We don’t understand what you’re trying to say with your semi-English.” And also lots of people asking me, “Hey, so you said I can email you. So sorry for being blunt and emailing you, but do you think it’s a good idea to use Trailblazer like this and that in our current company?” Actually people used that a lot.
I don’t know if it’s related to the publishing or to the chat channel we have now. Now most people just join the chat channel, and it’s incredible to see the activity on this channel. It’s so good to see that your stuff is actually being discussed by people other than yourself. I think people still feel this urge to participate in discussions with the author and other people, but now they use the official channels.
But I still get lots of emails of people telling me, “Hey, by the way thanks for writing this book”. Which is awesome. I mean, I haven’t had that in 10 years. I’ve been doing open source for 10 years, and you get an email every year, someone says, “Hey thanks for that gem”. And now you get lots of people supporting you and telling you that they like the book, or they find chapter seven pretty great, but chapter three sucks. Which is still better than no feedback at all. It’s incredibly cool.
Peter: It’s funny, I found that with my first book, Flexible Rails. I found that basically the better and the longer the book got, the less feedback I got. The book ended up being over 500 pages, but I found I got the most feedback when it was a couple of hundred pages, and I had real inflection points. I had to decide - should I put REST in the book or not?” I mean this was back in 2006, right? So yeah, if people see that you’re making this effort, and a community actually forms around the book, then you can get really good engagement from readers. It’s really cool.
Len: I actually have just one more question. When you were using Leanpub, or when you’re still using Leanpub, is there any one thing that if you could have your dream feature for us to build, or dream problem to fix, what would that be?
Nick: So - the process was already pretty cool when I started. I think someone kept helping me when I had questions, because you have this helpful like chat-to-one-of-our-people dialogue things. So a lot of problems got sorted by that.
My dream feature would be positioning images in flow text. Because you have to preview, and “Ahh shit, the resolution is fuzzy”, or whatever, or blurry. And then I had to play a lot with width settings. So if you had some way to position that in a “what you see is what you get editor”, or something. I know it’s a terribly hard feature to ask for. But that was the only real problem I had, was position. It wasn’t even a problem. I would just use float left, width 30% or something, and it’d work. But it took me like an hour or something to figure that out when I had that problem the first time.
Other than that, I don’t know. I’m more than happy with how Leanpub works. I mean and also it’s bringing this whole publishing thing to a new level. I don’t know how many new authors Leanpub has created in the last five years. How old is Leanpub?
Peter: We just turned six. We launched in April 2010, so yeah, we’re in Grade One now.
Nick: Happy birthday.
Peter: Yeah probably we need a cake or something.
Len: So yeah, we’ll thanks very much. Is there anything you in particular wanted to talk about or say before we go?
Nick: How many people work for Leanpub? I’m just interested in how the company looks like.
Peter: 5. We’re bootstrapped.
Nick: 5. That is great.
Peter: We’re a little bootstrapped startup, we’re self-funded, and we’ve been at this for a while. We’re technical book authors, right? So we built Leanpub basically like, myself and Scott founded it, and then Len and other people joined. And we’re trying to create the best way in the world to write, publish and sell books.
We’ve been, to date, focused so much on the writing and publishing aspect, and the selling and community stuff is underdeveloped. But where we’re going is, we’re going to be doing a lot of work around that. So we’re really interested in how books like yours have evolved, and where we could try to improve things in terms of community, or in terms of mobile. Right now for example, we don’t even have an Android app. But that’ll change. Reading on iOS will get better. So yeah we’re really excited, we’ve seen like thousands of authors use Leanpub…
So we’re super happy about how it’s gone.
Nick: I know so many people who started writing on it. I think when it comes to community, I think it’s wiser to integrate existing stuff like GitHub is trying, because you still have this comment on this book or something…. No one, I mean you don’t even see that it’s kind of like there. So I think I had like three comments. So I think integrating existing communities is the way forward.
One feature that would be cool to have - testimonials sortable.
Peter: Oh, manually ordering them?
Peter: Okay, that’s a good feature. Yeah. Right now it’s kind of random.
Nick: Right now it’s ordered by created at.
Peter: Yeah, yeah, yeah. I think we didn’t do anything, and that’s what you get.
Nick: The feature itself is great, it’s so much more authentic, genuinely, when you really have photos of people like, “Hey we use it and it’s great”. Or “we read it”. So that’s a cool feature.
Nick: Keep it coming!
Len: Thanks very much. This is obviously motivation for us as well.
Just before I go, I wanted to say, I think you’re the 6th Australian I’ve interviewed for these podcasts.
Nick: But the first without an Australian accent.
Len: Well that’s right. (laughter) I just wanted to remark upon that. It’s really interesting that some of our more successful authors have had this cluster who are living in Australia or come from Australia.
Nick: It must be the environment, the blue sky and the sun. So we don’t want to go out, because it’s too dangerous - you get sunburned and all that kind of stuff. You have to do outdoor activities, we don’t like that. So we go home and write books.
Len: Great well thanks very much Nick, we really appreciate it, and thanks for being on the Leanpub Podcast, and for being a Leanpub author.
Nick: Thank you guys for making Leanpub, it’s a great thing to do. Everyone should be using Leanpub.
Nick: Thank you.