Leanpub Podcast Interview #30: Alan Richardson

by Len Epp

published Apr 18, 2016

Dear Evil Tester by Alan Richardson

Alan Richardson is an independent software tester and the author of two Leanpub books, “Dear Evil Tester” and Java For Testers. In this interview, Leanpub co-founder Len Epp talks with Alan about that start of his career, the challenges of software testing and the reasons that huge software projects can result in disaster, and about his books and self-publishing on Leanpub.

This interview was recorded on March 24, 2016.

The full audio for the interview is here. You can subscribe to this podcast in iTunes or add the following podcast URL directly: http://leanpub.com/podcast.xml.

This interview has been edited for conciseness and clarity.

Alan Richardson

Len: Hi, I’m Len Epp from Leanpub, and in this Leanpub Podcast I’ll be interviewing Alan Richardson. Alan is an independent test consultant and trainer in Selenium WebDriver, exploratory testing, and technical testing. Having worked as a software professional since 1995, Alan has a variety of online training courses and maintains a number of websites - including compendiumdev.co.uk, seleniumsimplified.com, javafortesters.com, and eviltester.com. Alan has been a keynote speaker and has performed tutorials at conferences around the world. You can follow him on Twitter at @eviltester.

Alan is the author of two Leanpub books, “Dear Evil Tester” and Java For Testers. In this interview, we’re going to talk about Alan’s professional interests, his books, his experiences using Leanpub, and at the very end, ways we can improve Leanpub for him and other authors.

So thank you, Alan, for being on the Leanpub podcast.

Alan: Thank you. Hello, good to speak to you. Nice, professional introduction - thank you very much.

Len: I do my best, sometimes they go on a little long. I usually like to start these interview by asking people for their origin story, and I was wondering if you could tell us how you first became interested in software, and eventually in software testing?

Alan: Yes. So, I pretty much grew up with computers because the home computer started when I was about 12. So we got a home computer. Computing was one of the things I was really good at, so I went to university to study computing. My final year project at university was where I tried to pool in all the knowledge that I was really interested in. So, my final year project was a COBOL interpreter - because we still did COBOL in those days; an Ajax-instructed programming diagrammer, because we were trying to construct our programs without actually getting our hands dirty and doing any programming. So I combined the two and created a diagram tool that I could debug COBOL applications in.

Then when I left university I started writing test tools for a testing consultancy. When you work in a test consultancy, you start doing testing on site, because they can make more money out of you that way. And so then I started learning about testing. I read as much as I possibly could about testing, so that I could be a test consultant. Since then I’ve done testing jobs as a tester, test manager, head of testing.

And at one point I decided I deliberately needed to learn how to work within an Agile project, and test web applications. So I geared and moved towards that. I’ve been testing and talking at tutorials and conferences, and now I’m independent - primarily around Agile automating web apps, APIs, and helping people improve their technical testing process.

As you mentioned, I’ve got a bunch of websites and I’ve put a lot of material online that way. I’ve pretty much just grown up doing programming and doing testing at the same time. Even when I’m managing, I’ve kept my technical skills up to date, because for me, we’re software professionals. So programming, testing - it’s all lumped into the same thing, and that’s what I like doing.

Len: And at what point did you start reaching out to teach people about testing, and to train them?

Alan: So that probably was just a side effect of, because I was talking at conferences, you then extend that into doing tutorials. I started talking at conferences, simply because I was hearing so much rubbish at conferences, that I wasn’t really prepared to pay to listen to it. So I was only going to go to conferences if I was talking at conferences. Then I could put my point of view across. And then that was just an extension - the more you do that, the more people ask questions, the more you hear that there’s gaps. And just generally, because you’re building up your skill set, and because you move into management, you’re starting to train your staff. So you get in the habit of training your staff, pairing with your staff, improving people’s technical skills. And just as a side effect, you start putting more material out there.

Len: And what was the form the rubbish took? What made the rubbish “rubbish”?

Alan: I mean, well, this is a testing industry. Computing continually renews itself, right? So there’ll be people listening to this, people reading books, that have never seen anything except Agile. But in the world before Agile, when it was all Waterfall and very structured, there are a lot of people pontificating on how to do things. We had projects that would go on for two or three years, and never release anything - and get cancelled. But people still knew how to do testing under those environments.

It was things like that - very structured processes, where we have to document everything. Everything needs to be at the level where anyone can do it. There’s specific phases, specific templates, specific documents. And it wasn’t really about the skill sets that people needed, it was about the process that sat on top. I’m much more focused on the skill sets, the approach to testing that individuals need to take, and how we coordinate and fit ourselves into processes in very unique fashions, rather than huge structured processes and very formal ways of doing things.

Len: That’s really fascinating. So there was a parallel in the testing world of the shift from, say, Waterfall to Agile?

Alan: So I think - even with the Waterfall projects we wanted to have testing add more value, and do things more quickly, and avoid this split between testing and development. Because with Waterfall it was very much a room of developers, or a building of developers and testers in another building, and communication entirely by documents, rather than communication. We would be building things, like test scripts, just to keep ourselves busy, because we knew that at some point, the software was going to drop and we had to test it. And it would be filled with issues, and if we missed them we’d get blamed.

I mean, on an Agile project now you see people perhaps building software for five days, or four and a half days, and then passing it over and expect it to be tested in half a day. For a bad Agile project, those are things that happen. Same thing would happen on Waterfall. People would build it, they’d build it for two years - and then drop it, and expect it to be tested in 3 months. So you get similar kinds of percentages, but it’s simply bad processes don’t have the communication that people need in order to make them effective.

Len: That’s really fascinating. My last interview was with David Greenlees, who wrote a book called Software Testing as a Martial Art, and he tells this great story in the book, about how his first job was testing - I believe it was part of a government program. When you say you had to document things - he had to document when he went to the bathroom, because of the time it would take. And it seemed like, in those olden days, people were trying to put a kind of industrial management model first - and then stuff software development and testing into it.

Alan: It’s hard to say exactly what the rationale and thinking process behind it was. I think, a lot of the time, with any process, you have managers that don’t know how to do the work. So what they know how to do is manage. If you’re trying to extend the time it takes to build a project, you want to get it right. So you spend a lot of time in the design phase, trying to gather the requirements, make sure they’re correct. If you wait, then you’re never gonna get it built.

So you spend a lot of time doing that, and then in parallel you want to start developing it, but you’re not putting anything out into production or testing - because it’s not fully finished yet. And it just takes a whole bunch of time. So people want to protect the process, and make sure everything is connected, make sure the requirements are listed into the test cases so that you know exactly when things are done and when they’re not. And it’s just, if you take a long time to do it, it doesn’t work as effectively.

Len: Yeah, yeah. That’s a very great line about how managers often don’t know about the work that they’re managing, but they know how to manage. I can see a lot of probably not-so-happy experiences behind that. I was wondering, actually, you said - did you study computer science in university?

Alan: Yes.

Len: That’s interesting. I would say of the people I’ve interviewed for this podcast, maybe less than half did so - of the people that ended up as being involved professionally in software development and testing. I was wondering, if you were starting out now, do you think that you would go to university again?

Alan: I don’t think I would probably go to university. I would still study computer science. I think computing now, there’s so much information out there, most of it, free, some of it you have to pay for. And the books I studied, and study, that I get a lot of value out of - are the old books from the 70s that we grew up with, the 80s. And some of those are really hard to find, but some of them you can get a hold of really cheaply. They still add a lot of value.

They were essentially trying to think through the process of software development. So when you read those, you see them saying things like, “Well we tried this. We think this might be an issue here. We’re going to try this part, and then there’s another way of doing a structure programming. And we’ve tried this way, and tried that way.” We’ve forgotten some of the thought processes; now we’ve codified them into “Thou shalt do it this particular way”.

But there’s so many ways now - all the languages are free. We now have computers. You can buy a computer, for like 100 quid or something, that is way more powerful than we ever had at university. It can run C++ compilers, and you’ve got JavaScript, you’ve got programming languages in your browser now that can do…. I mean I see people writing emulators in JavaScript for the computers that we had when we were younger. I mean, it’s an amazing amount of power, there’s an amazing amount of knowledge.

What’s probably more important now is that people build a portfolio online. And we’ve got ways of building up free blogs. You can put all your code in GitHub. There’s just ways of communicating to the outside world what you have been learning, that they can pick up on when you go for a job. And rather than spending five years - as I did, at university - if you spent 5 years building stuff in lots of different languages, trying to create different programs, trying to solve different solutions, different needs, if you’re studying Computer Science as you do that.

Because a lot of the things that we – when I’m doing testing, when I automate work, I still draw on the lessons I learned from creating compilers and interpreters. I mean, I’m still building abstraction layers, which are essentially verbs from the programming languages that are doing certain things. And also, that ties into the adventure games I used to write and play as a kid. The text adventure games - where you’re building maps, you’re exploring an environment. You’re writing a command line interpreter that deals with natural language, you’re parsing things. Even in a very crude way, those are computer science techniques that we still use today.

So I probably wouldn’t go to university, but I would certainly still study the topics that we did.

Len: That’s interesting. Pardon the tangent, I like adventure games very much myself. Do you still play them?

Alan: Periodically. I mean, at the moment I’m writing a multi-user dungeon adventure game that I can use for training people in testing.

Len: How does that work?

Alan: So that works on the basis of it’s a mod, it’s got very simple GUI, but over the course of the training, the GUI starts to degrade. So you get less and less functionality exposed by the front end. So when you come in it will say, “You’re in this room. You can go North, South, East, West.”. You can click on the North, South, East links. And then the links go away. So you have to either have remembered the URLs to use, or you have to know the techniques for navigating around. You have to look in the source to inspect it, to see what ID’s there are for the things that you can pick up. There might be stuff hidden in the source code on the browser that tell you whether it’s a good thing to pick up or not. There’s an API that underpins it, so eventually it kind of forces you in the training to drop down below the GUI and start interacting with the adventure game just with the API.

Len: That’s really fascinating. Do you have an audience for that already?

Alan: I’ve used it a little bit. And I’m just going to be starting to use that in my training courses when I do them face-to-face for the exploratory testing work.

Len: My next question, actually, is about testing. I was wondering if you maybe have a war story of some kind you could tell us about? Maybe the most difficult testing job you ever had, or perhaps the most difficult problem you’ve had to overcome in your career so far?

Alan: Just one war story - that’s the hard part. If I think back to the worst project I’ve ever been on, the worst one - I’m not going to name company names. When we went for the interview, we went through the interview process and I said, “You’re going to definitely need at least two people to do this work”. When I started on the job, it was just me. And what they wanted me to do was build a performance test harness for some code. But they wouldn’t let me see the development code, because it was top secret.

I had to essentially use their Java libraries that they were building as part of their code, without seeing the code. So consequently, they would just give me the .jar files and not tell me what they were doing. I had to build a performance test harness around that. So I used a lot of my complier creation skills in order to build a front end for that. But when things went wrong, I had no ability to debug it because they wouldn’t let me see their code.

And then it got to the point where I thought we were finally going to work with the development team. But that was the point at which they were starting to outsource all their testing to India. And the reason we were starting to work with the development team is because they wanted to get rid of us, and they wanted that knowledge transfer. But that was also one of the environments where - when we started in the testing, we were in an entirely different building from the development team. And we had to push hard to actually be co-located. So they eventually moved me over so that I was sitting at a corner in the same building, but far enough away from the developers, so they wouldn’t notice I was there.

Len: This has come up a couple of times. Why is it do you think that there was - or perhaps still is this desire to keep developers and testers physically separate?

Alan: I mean some of it is - they’re almost in different departments. You’ve got the software development department and the software testing department. Different budgets, different managers, different hierarchies. The developers don’t really get to speak to the users. They get documentation dumped on them from the analysts. The testers are expected to speak to the users, because they can kind of bridge that. No one wants to speak to developers, because they speak tech.

The testers can speak business, and hopefully a little bit of tech - so that they can bridge that communication gap. I mean it’s really hard to know what the thought process around that was. But for me, it’s hard to know what the thought process behind an open plan office is. Where people are sitting there, and you can hear everyone and you can’t get any work done, because it’s so noisy. And there’s a lot of decisions that get made in the business world that I don’t understand.

Len: It’s curious, the open plan office is something that was, I think, in some business literature, presented as an opening up, but from another perspective, it’s sort of a Benthamite Panopticon, where everyone can be viewed all at once, and you can make sure that all your beavers are busy, beavering away.

And then work becomes a kind of performance, where you try to look busy all the time. Because, especially for things like software, the work is happening in your head. But there’s a certain kind of observational management practice that doesn’t know what to do with someone sitting there thinking.

Alan: Yeah.

Len: Actually on the topic of what things look like to people, you’ve got a video on YouTube I found, where you talk about the importance of thinking visually in software testing, and the drawbacks of not thinking visually. I was wondering if you could explain why thinking visually is so important, and, separately, why not thinking visually is such a trap?

Alan: In that video where I’m talking about thinking visually, I’m essentially talking about making what you are thinking visible to other people. A lot of people, when they’re talking about thinking visually, they’re talking about diagrams or mind maps or whatever else. I’m essentially talking about making your thought processes visible to other people. And mind maps can help with that.

But I think - I’ve been using mind maps for a long time, and when I used mind maps at university, it was for me, to trigger my memory. A lot of times, people now use mind maps more as an outline, rather than something that conceptualizes what’s in their head. So sometimes a really good mind map works for the person that built it, but often it’s a very bad communication tool, because it represents the mind that has created it, and no one else understands that mind.

So when mind maps are used as a communication tool, they’re much more a structured outline of stuff, which hopefully people will read. But that’s not necessarily a great mind map, because I built mind maps so that I could have a very concise drawing that I could put down before I answered a question. And then I would use that mind map to trigger the thought processes I would use in answering that question.

Whereas now, a mind map has to be a very, very detailed thing. Because we’re using it for communication. I like to draw diagrams of graphs, so, paths through systems. And that for me is a very visual way of describing flows through applications, and I can assess coverage and things like that.

The more that we learn how to visualize information, we start to understand - hopefully - that underpinning all these visualizations is a bigger model. When I draw a graph of a system, that’s not my full understanding of that system. That’s just me describing that system as a set of executable paths. There are other things that make up that model, like requirements, the data that’s in there, all these other parts. That graph is just a projection of part of the model, and consequently it loses a lot of information in that projection. So I need other models that also make visible the other parts of my bigger model.

I mean it’s kind of like Chomsky with his linguistic process - the universal grammar, where he’s talking about how you have a deep structure, a deep model of the world, and you surface structures, different sentences that represent that world. None of those surface structures, none of those sentences describe the full scope of the world. When you construct the surface structure, there is emphasis put on one part of that structure at the point that you’re communicating. There are things that are missed out. There are things that are confused. And we could put the wrong word in. We state something incorrectly because of that transformation process.

So what I would like people to do is be conscious of the models that they have, and one way of doing that is to try and visualize them to some extent. Then you can start to analyze them for coverage - whether you’ve understood them, whether you were miscommunicating them. You can look at them for gaps. You can look at them for risks and issues of what you’ve missed out and what you haven’t. And you don’t necessarily need it to be visual in other terms. It could be a list of statements - that’s a visualization of that underlying model that you have of the system.

Len: Do you find when you explain this to people, does the light bulb immediately go off, or is it something that requires really getting them to understand the value of diving into learning a new approach?

Alan: Most people don’t really grab hold of that. I mean, essentially what I’m describing, when I’m describing that, is systems thinking. I’m describing that there are underlying models and we have interconnected systems, and there are different ways of representing a model of that. It’s something that we essentially have to study longer term. I only ever tend to explain things in that kind of way when I’m doing a talk. If I’m appearing with someone or working with someone, what I’m really doing is looking at an individual person, how they’re thinking. And I’m trying to understand the gaps in their current thinking at that point in time.

So we might have a way where we try and get them to write down in advance what they’re going to do before they do it. Or we might have them try and write down what they are doing as they are doing it, so that we can then look back and assess what they’ve done. If they’ve done a couple of hours of work, we might try and summarize it in a diagrammatic form, so that they can better review it. When you’re working on a much more individual basis, it’s much more targeted process. You very often don’t have to explain the overall process and model.

Len: That sounds really effective. Obviously training people in thinking at a kind of meta level about testing is very important to you. For anyone listening who might be starting out in a career in testing, what would your top bit of advice be to give them, as they look out ahead at maybe a couple of decades of work?

Alan: Well, so if they’re just starting out, they have to read everything and disbelieve everything, right? Because we’ve spent a long time writing a lot of stuff about testing, put a lot of processes out there, created a lot of standards, a lot of templates. But none of that will help the individual, because they have to implement it. And they have to justify everything they do. So if they are on a project, and they’ve gone, “Yeah, we’re going to do IEEE7612”, or whatever the template is, and then they start putting in all this boilerplate information that’s of no value to anyone, they have to be able to justify that, and they can’t justify it in terms of, “We’re doing this, because that’s what this template says”. Or, “That’s what this book says.” Because they lose credibility. They have to almost immediately take responsibility for what they’re doing. And that’s hard when you don’t know what you’re doing. So unfortunately, you have to try and learn as much as possible. But only do the things that you see real value in.

What I’ve been doing over the years is, I’ve been slowly taking things away from my process. You start by putting in all the templates and everything else, but then you go, “Well what happens if I don’t use that template and I just have a couple of pages that describe what I’m doing?” Then you discover that if people don’t understand something, and it’s important to them, they will ask a question. And then you see whether you’ve missed something out in your communication. And then you put it in.

So what I tend to do now is, if I have to create documentation, I’ll create documentation that helps me think about what I’m going to do, and why I’m going to do it. And if when I use that to communicate to other people, there are gaps, then I can either address that individually for them, or add it into the document - if it’s a documentation-based process - to help other people understand what’s going on. But fundamentally, testing, or anything, when you’re learning it, you have to take responsibility for it.

Len: That leads me right on to my next question, and shifting to your books. In the promotional video you have for “Dear Evil Tester” you talk about testers building up the right attitude. You’re very explicit about that, and I was wondering if you could explain your thoughts about what the right attitude is?

Alan: Sometimes that attitude has to flex. Testers have to be really flexible in how they approach projects, because most people on projects – other roles don’t have to have quite the degree of flexibility, right? They will get away with more stuff. Programmers will get away with more stuff, because they are so utterly important to that project. You could get rid of almost anyone on the project, except the programmers - if you want to actually create software, right? But if you get rid of the programmers, you don’t create any software. They’re vital. They have a huge power base from which to work.

Testers are often sidelined, pushed apart. So we have to have an attitude of flexibility, so that we can fit into the gaps. We can spot risks anywhere. We can communicate to anyone. We have to have the ability to feel that we can go anywhere. Not just in the software, but in the project. We can go and speak to the designers outside of the planning meetings to get more information to give us an edge. To speak to the programmers before they write any code, to help or pair with them or whatever else. It’s an attitude of doing whatever it takes. Because sometimes on projects, other people stick to well-defined roles, and don’t look outside them to see what the risks and issues are.

We, as testers, are expected to communicate risks and issues. We have to do whatever it takes to find them and identify them. Sometimes we have to say things that no one else will say. No one else is prepared to speak up and say the things in meetings that we say. We have to be prepared to be shouted down, to be locked out of rooms, and break our way into those rooms to - whatever. I want an attitude where people don’t sit back and take it, knowing that there’s risks and not speaking up. That won’t help.

Len: Yeah that reminds me of a huge question a layperson might have, a curiosity they have about software development, that comes from their experience with dramatic failures. In the United States in the last couple of years, the most prominent one being the Obamacare website rollout. But you don’t have to be in the United States to have encountered that. We’ve had - I’m speaking from British Columbia in Canada - across the country, especially in the province of Ontario in the last few years, there have been some huge catastrophic failures. I’m sure this has happened in the UK as well, and I was wondering, if you were standing in the pub talking with someone, and they said, “How is it that you guys can all be in buildings for years, and fail?” How would you answer that question?

Alan: Well that’s easy. I’m surprised more people don’t know how to do that, because we know how to do it. It’s a well documented approach to knowing how to do that. Many software development books tell you exactly how not to do that. It’s a very structured process. There’s a lot of management books that tell you exactly how to achieve that - by not allowing people to communicate, by putting rules and things in place.

But unfortunately, when you’re in the pub and you’re talking about it, and people say things like, “Why didn’t the testers pick up on those kind of things?” - they might well have, right? Unfortunately you don’t get to see inside that project, to know how much turmoil was going on. To know who was sacked. To know what risks were raised. To know what risks were completely ignored. To know what was delayed, so that no one had any time to work on it until it went out. To know what the politics of communication were.

Because really, fundamentally all these problems arise because people are involved. And people take personal affront at certain things. People want to control their power base within a department. People fight over budgets. People have to maintain their budget, so they’ll use it up hiring their staff, and not leave any to anyone else, right? It’s much less I think about the project, and more about the politics in the organization and who’s involved. And sometimes with these big projects, you have competing consultancies in place.

Rather than building it internally with their internal staff, they’ve brought a bunch of consultants in. Then there’s communication links, because the consultants aren’t just working for the client, they’re also working for themselves. They’ve got a power base to maintain. There’s contractual obligations. So people stick to the contract, rather than doing what is actually required - because the contract was built before the requirements, and nothing changed, so it all drops back to the contract. So if you really want to make things break, spend a lot of time negotiating a really tight contract that is inflexible. That will almost guarantee a project to fail.

Len: That’s really interesting. That reminds me of someone I know who was casting about for a job some time ago, and got a job as a project manager on a big software project - and arrived at work to discover he had nothing to do. His job existed to keep the budget up. That’s interesting in itself, but the lived reality of it was so sad and humiliating that it couldn’t– I mean, my friend never said this to me, but I could only imagine how once the edge of that wedge is in a project or a workplace, that kind of problem must spread up and down the command chain.

Alan: Yeah, and it’s hard when you’re in those positions. Because you’re desperately trying to do the right thing, but the system is against you. People aren’t listening to what you’re saying. And it’s hard sometimes to change a system from within. But people stick with it for a long time, because you think you’re making a difference. You’re going to the meetings, you’re saying the things. It looks like something’s happening. You’ve made an influence somewhere else, but it’s taken you months to get the point where that person now exhibits doubt about what’s going on.

And then they go and see their manager, and it stops almost immediately. So you put two months of work into that strand to try and change it. That didn’t work. And you start going to speak to this other group over here. Sow the seeds of doubt, try and get a rebellion going on in this corner. That seems to be starting up. And then after nine months, you might go, “Right, I give up. I can’t make the changes here”. But there’s other people doing that at the same time. And over a long project, it’s very easy to see how it just slowly crumbles and falls apart - despite a lot of people trying to fix it.

Len: Well that’s very depressing.

Alan: It can be. And that’s why the Agile process is really, really good. They still have problems. They still have communication problems. They still have politics. But they’re much more focused on building something, and they’re much more focused on getting something out the door, and having people bypass the politics - they have that much smaller political structure within there. So it’s a lot easier to get things done and out the door when that’s happened.

Len: And for anyone listening who’s not perhaps familiar with what Agile is, I was wondering if you could explain a little bit of the secret sauce, and why it works the way you just described?

Alan: So Agile can be as bad as Waterfall, because when you come into a software development team, and they say, “We’re doing Agile”, you still have no idea what they’re doing until you see it. There are so many different ways of interpreting that. But I think Agile works best when you are releasing, into production, regularly, software where you have put a lot of effort into having a lot of automated validation around it, to try and allow you to move fast. Where you’re doing a lot of exploratory testing, you can do a lot of exploration around it, and you’re trying to build it in very small chunks that add value, and where you’re trying to release not necessarily fully formed chunks of very pretty interfaces, but functionality that meets a user need -if it’s an internal user, you can cut corners on the design until later, because you’re trying to make sure that is actually what the person needs.

But sometimes people say they need something, and the don’t actually know that’s really what they need until they see it implemented. And then they go, “Oh wait a minute. But it doesn’t do X or Y, because we didn’t realize that was important”. Or, “We didn’t realize that wasn’t conveyed by what we said”. And that’s why if it’s a really long project, they don’t find that out until the very end, if at all. But on Agile projects, they can hopefully find that out very quickly.

Len: On the subject of evaluation of progress and status, in one of the questions that’s asked in your book, “Dear Evil Tester” - which takes the form of a kind of agony aunt column where people write in questions about problems that they’re having and then you sort of write these humorous replies - one of the questions asked is, “What is the most evil metric?” And in your answer you say, “There are many to choose from, but one to bring up is number of test cases.” I was wondering if you could explain a little bit about what that metric is, and why it’s most evil?

Alan: So that goes back to the early days in projects, where we were very Waterfall. Waterfall is about - we’ll create our designs, we’ll create all our requirements. Then, what the testers have to do - while all that’s going on - in order to justify their existence, is create a lot of test cases, right? But those test cases have to be cross-referenced back to the requirements.

Now, we have no measure of how productive the testers are at that point, because they’re not testing anything. So we have to count the test cases to know how productive the testers are in terms of creating what we’re going to test. Testing is infinite. We can do so many things. So if we start and codify it into each test case, we’ve always got an infinite number of test cases we can come up with. And so we end up creating more test cases than we could ever possibly run in the lifetime of the entire universe - let alone the three months that we’re going to get to actually execute this stuff.

So we have to count test cases, beccause then we’ll know how much stuff we’re going to do on the system. But I mean, that also goes back to - I was working on a project in London. I was relatively junior at that time. But I had to report on progress of testing. And we had tools. So we had tools where you put in test conditions, and they’d be cross-referenced to requirements. We’d write test cases, we’d write test scripts for those.

And I wrote a metrics database, and it was brilliant, because it was so flexible. I could analyse the underlying tool, count how many test cases, count how many test cases were cross-referenced to requirements. I could count out how many test cases each tester had written. How many still had test scripts to go? It was a work of genius. I had at least 100 different metric queries that were running against this thing. It was overloading people with information. Because it wasn’t information, it was just data. And it was so flexible, it just allowed me to pick up more and more stuff and that felt like we’re making progress.

But really we weren’t making any progress. We had a change control process. There was new specs, new designs. We had to do a lot of cross referencing, because otherwise you don’t know, out of all this stuff you’ve written, what you have to change. It’s such a wasteful process. When lean started getting promoted within the software industry, that was really useful, because it’s made the point - if you’re producing something, and it’s really not being used yet - it’s essentially waste. You’re building up a backlog.

We would build up a huge backlog of test cases. We didn’t even know if they were correct. If you tried to execute that test case against the system, apply that test case to the system, it probably wouldn’t work, because the implementation was different from the requirement. And all you’ve really discovered is: we’ve got a gap. But we’ve now got a huge set of scripts and cases that we have to rework because of that. And it’s just hugely wasteful.

Len: I was wondering if you could explain a little bit about where the idea for “The Evil Tester” came from? And in the context - and I’m sure you get this question all the time. What does “evil” mean?

Alan: The Evil Tester character was a result of frustration. I mean, when you talk about Waterfall projects, you talk about a lot of frustration in the process. So he was there as my little doodles in some of the most pointless meetings ever, to vent frustration. And he would be stabbing project managers or whoever else was causing me most anguish at that time.

And the notion of evil. People very often get hung up on specific words. So I want flexibility in the words that we use. When you bag something that’s evil, it typically doesn’t get at the core of the problem, right? Because you’ve just said it is evil. We don’t understand its motives. We don’t understand the motivation. We don’t even really look at what it’s doing. We just classed it as evil, it’s bad, it’s wrong - whatever else.

Also, we read here on projects that testing was a necessary evil. No one ever wanted to invest in testing. Testing was just something you had to do, because if you didn’t, it wouldn’t work. It was necessary. No one liked it. No one liked having to have lots of testers, because testing would inevitably overrun. They’d find problems. I mean, how dare they find problems in our perfect system before it went out? Then we’d have to re-code it. Then the testers would have to redo their testing - and we wouldn’t know what the impact was. We’d have to do all the testing again. We’d have to re-cover everything. So it was just this necessary overhead.

And rather than actually fix the system of development, and make the system of development work - we viewed testing as this necessary evil. I just kind of thought in my head, “Well if testing is a necessary evil, then evil testing is something that we necessarily have to do to get at the heart of this”. And it was helping me build an attitude of, rather than me having to stick in this corner, I should have the ability to go and speak to someone.

Even though that’s not allowed. I’m not allowed to walk over there and speak to those people, I’m going to do it anyway. I’m not allowed to see those designs before they come over to me signed, but I’m going to do that anyway. So it was about giving myself permission to do the things on the projects we were not allowed to do, but which were necessary to allow that project to recognize the problems that it had early.

Len: That’s a really great explanation. And I think it ties in very well with some of the themes of what we’ve been talking about, about how it’s the people and the rules and perhaps the management practices, and the roles that people play in their internecine project or office politics that are really in the way of getting things done.

I was wondering actually, switching to your role not as tester, but as a self-published author, I know that for your book, Java for Testers, you actually started publishing it before it was finished.

Alan: Yeah.

Len: I was wondering if you could explain what that process was like for you, and why you chose to do it that way?

Alan: So the whole self-publishing thing has been an experiment for me, from day one. I wrote a book before I started doing things on Leanpub, I did a book called, Selenium Simplified. I did two editions of that and I learned a whole bunch of things doing that. And I thought, I am never going to write another book with code in it, unless I have the ability to automatically extract code from my code base. That was a huge pain, copy and pasting things. And when you read a lot of programming books, you see errors in the code. And you think, “How on earth could this happen?” I never wanted that to happen again, so I built a macro generator I could put in my code. That will automatically extract the code, and stick it in the text file, so that I could incorporate it into Leanpub.

But part of the reason for doing it in chunks was - it was kind of an experiment where I thought, “I’ll see if people actually are interested in this”. You get it out there, you start to build up a base of people reading it - checking to see whether it’s any good or not. And you get feedback that way. I think there are some types of books where that’s really important. If you have a really good outline before you start, that’s a really good process, right? Because people know in advance what’s coming.

Part of the issues for me was, because I was building it, people were putting in suggestions. I would then incorporate the suggestions. So the book was like double the size that I actually wanted it to be, because people kept asking for more and more stuff. So it was good, because I’m meeting the wants that people had. And there were people that were invested in this book. But it made the whole process a little bit longer than I wanted. But I mean it’s a great process to get feedback, to see it building up incrementally.

But also there’s a kind of debt in the middle, where you haven’t finished the book yet, and you’ve got other work to do. I’m sure there’s a gap of 3 or 4 months where I didn’t touch the book, and I felt really guilty, because people had bought this half finished thing, that I wasn’t actually even attempting to finish at that point. But it was there in the back of you’re head. So you’ve built this monkey for yourself that’s walking around all the time nagging you, “Finish it”.

But the hard part of writing any book is finishing it. And if you incrementally go through it, you’ve got that guilt motivation, because people have paid for it, and they really deserve a finished product. And you’ve got the benefit that if you haven’t told them in advance what you’re going to build, you could just stop and say, “Yeah I’m done. Finished, that’s it”. But I didn’t really want to do that.

So I like the notion of incrementally building it, but I’m not sure I’d incrementally build it to such an open ended goal again.

Len: I see. Did you have people actually nagging you during that sort of…?

Alan: No, it was just me. But I had questions coming in, like “Will you cover this?” And you go, “Well I hadn’t really thought I’d cover that. But okay, I’ll cover that and that.” So then you go home and you have to just buckle down, get it finished. Even if you’re doing it incrementally.

Len: I noticed that the promotional videos you have for your books have some pretty well made cartoons in them. I think in fact for Java for Testers I think the whole video is a cartoon. I was wondering, for the benefit of any self-published authors listening, or anyone who’s thinking of going down that path - how did you get those cartoons made?

Alan: Giving away all my secrets, that’s what you want.

Len: You don’t have to, you don’t have to tell….

Alan: No, that’s cool. There’s a website called powtoons.com, and it basically lets you create these animated things on your own. They have a bunch of templates. Qhat you do is you create a voice over, you upload the voice over, and then you create 15 short chunks of animation. And it wires it all together for you. It’s pretty easy to use. So it’s really not as hard as it appears.

Len: Well thanks for that, that’s a really great tip. Also, I noticed there’s a print version of Dear Evil Tester and Java for Testers.

Alan: Yeah.

Len: Both books I believe are for sale on Amazon. I was wondering if you could explain a bit about what your motivation was to make print copies, and what that process was like?

Alan: So the motivation is. people ask for print copies. I mean I’m looking at my office now, and I have just hundreds of books in here. I love print books. And particularly when I’m learning a programming language, I’d much rather have a physical book to learn from, because it makes things easier. But I like digital books as well, because I can read them really fast. I can plow through them. I can refer to it, I can carry it around in my tablet. So depending on what we’re doing at different times, it’s a massive convenience.

Leanpub is really useful for creating both the digital books and the print books. Because you’ve got the ability to create a print-ready PDF. That print-ready PDF, pretty much you just upload straight into CreateSpace, and all you have to do is add the cover. The hard part is adding the cover. You do nothing to help people with the cover. But there’s really cheap desktop publishing tools you can get. Build a cover, and export it as a PDF. Join the two together in CreateSpace, and you’re pretty much done at that point.

The hard part is - when you’re editing the book yourself in digital, you’ll find a subset of errors. As soon as you go “print,” then you’ll find other ones. You will see other errors in the print copy than you do in the digital one, just because it’s a different medium. You’re reading it a different way. So for me, it’s also very important to go, “print,” because I will see different problems.

Len: I’ve had that experience - I think we’ve all had that experience where you write something on your computer, and it’s your baby and it’s perfect. And the second you press, “print,” you’re like, “Oh there it is, right in the title - a typo.” And there is something about switching mediums that just makes things suddenly apparent that weren’t otherwise.

Alan: Yeah, but it’s nice - it’s so much easier now.

Len: So the process by which you test the print version is that you upload the print-ready PDF from Leanpub to CreateSpace, or whatever service you’re using. You upload the cover that you’ve designed, according to the dimensions of the book size you’ve chosen. And then you order a test copy. Is that correct?

Alan: So the first thing I do is I print out the print-ready PDF. I have my little laser printer here. I usually do two or three printouts, just locally, so I can read it on A4 double-sized. I’ll then upload the print ready PDF and get a proof copy through either CreateSpace or Lulu. It depends what I’m targeting.

If I want a physical product to see what it looks like, beccause I think this is it, then I’ll be looking for the CreateSpace one, because that’s going in Amazon. If I want something fairly fast, because I want a different medium to check it on, then I might just get the printout from Lulu. Both of them have the same process for pretty much uploading. It’s just that Lulu deliver a little bit faster than CreateSpace do in the UK.

But before I go live with Amazon, I’ll make sure I’ve got the CreateSpace proof copy, because you also need to get the cover aligned. You see so much stuff bleeding over the spine, because you didn’t get the dimensions correct. There’s a lot of tweaking. And that process actually takes quite a long time, because you upload it and CreateSpace go, “Right, we will double check this”. I don’t know if they do it manually or automatically, but that takes about a day. Then you have to wait three or four days for the thing to be delivered. So you’ve got that lead time on each proof copy. You really have to get it right.

But it was only recently that I discovered the diff-pdf tool, where I can compare the two print-ready PDFs. Like the one that I’ve just sent to CreateSpace, and the print-ready PDF that I’ve downloaded from Leanpub. I can diff them to make sure that the changes I expected are in there, and there’s nothing else going wrong.

Len: Yeah, that’s great, diff-pdf. I read about that in your blog post about the process. It sounds like a great, a really useful tool.

I guess I have just one more question for you, while I’ve got you here. If we could build a magical feature for you in Leanpub - if there was something that you saw wasn’t working right during your process, writing your books and publishing them, or if there was just something that was missing that you think would make a great addition, what would that be?

Alan: So most of it is pretty good for me. The only thing that’s different is, when I was writing the Selenium Simplified book I started that as a word processed document - the first edition was word processed, that was horrible.

The second edition, I kinda created my own version of Leanpub. My own version. It took XML files and generated LaTeX. So I had a lot more flexibility. And what I found is, LaTeX has the really good facility of writing tiny macros before you do the final run-through of the tech process, that can make a massive change in the formatting. It’s kind of mini-CSS changes that make a massive difference. So just a little bit more control over the LaTeX, or even seeing the LaTeX, so that I can format it myself. But I mean, that’s a bit more techy than most people are ever going to want to ever use.

I’m assuming that you use LaTeX under the cover just because of the way that it’s formatted. But just a little bit of extra tweaks, because sometimes I think, “Yeah, I would really like the font to be just a little bit smaller than the one supported. But I don’t mind that much”.

Len: Okay, well thanks very much for that suggestion. I’ll pass it onto Peter and Scott. There’s actually a really sort of deep issue at stake in that kind of thing. For optimizing the process of writing and publishing for people, we’ve found, because we’re tinkerers and perfectionists ourselves, that formatting can become a distraction.

Alan: Yeah, it can.

Len: And as Peter Armstrong, my co-founder puts it - formatting is procrastination. You can often find yourself tweaking all this formatting, when you know you’ve got a chapter to write.

Alan: Yeah, that’s why it’s only necessary for the print-ready PDF part. Because for the ebooks it’s fine.

Len: I see.

Alan: It’s only when you’ve actually got to print, and you just want a few little possibly extra tweaks. Again, it’s perfectionism for no real value, because Leanpub has all the features that add the value in the print process. This is just that little extra bit. And it’s completely unnecessary. But I had to answer your question.

Len: No, no, thank you. These things are very important. And it’s interesting, because I think it wouldn’t be at all inconsistent with our principles to have some more formatting options available at the print-ready point. Because that’s when - so the other thing my co-founder Peter Armstrong says is that, “Formatting is procrastination until the book is complete - completely written.’

Alan: Yeah.

Len: “And then it’s formatting time,” right? Which is why we have InDesign export as well.

Alan: Yeah.

Len: Thanks very much for that suggestion, and thanks for being a Leanpub author - and thanks very much for this conversation. I really appreciated it, and learned quite a bit.

Alan: Thank you, and thank you for putting Leanpub out there. because otherwise I’d have to maintain my own system, and that would be a complete pain. Leanpub has a massive amount of value in it, this publishing process.

Len: Thank you very much.

blog comments powered by Disqus