Dmitry Vostokov, Author of Encyclopedia of Crash Dump Analysis Patterns

Len: Hi I'm Len Epp from Leanpub, and in this episode of the Frontmatter podcast I'll be interviewing Dmitry Vostokov.

Based in Dublin, Dmitry is a speaker and writer and founder of the Software Diagnostics Institute. He has written over fifty books on a variety of subjects including memory dump analysis, software and memory forensics, malware analysis, and more.

You can follow him on Twitter @DumpAnalysis and check out his website at dumpanalysis.org.

Dmitry is the author of a number of books published on Leanpub including Encyclopedia of Crash Dump Analysis Patterns: Detecting Abnormal Software Structure and Behavior in Computer Memory, Third EditionDetecting Abnormal Software Structure and Behavior in Computer Memory, Third Edition.

In this interview, we’re going to talk about Dmitry's background and career, professional interests, his books, and at the end we'll talk about his experience using Leanpub to self-publish his books.

So, thank you Dmitry for being on the Leanpub Frontmatter Podcast.

Dmitry: Thank you for inviting me to your podcast.

Len: I always like to start these interviews by asking people for what I call their "origin story." So, I was wondering if you could talk a little bit about where you grew up, and how you got interested initially in computers and technology?

Dmitry: Oh, that's a great question. I grew up in the Soviet Union. I went to school just before the so-called perestroika, if everyone knows about that.

I got interested in many subjects. I was interested in history first. And but then - around, I would say, when I was maybe 12 year old or something like that, I got interested in chemistry. So I started participating in Olympic Games, that were very popular in Soviet Union at that time. I was a winner couple of times at different levels, but didn't make it to international level.

As a result, I entered Moscow State University. But due to some small, I would say - fortunate probably, because - a fortunate accident, I was enrolled into a special computer group. It was chemistry with some computational chemistry.

And before that, actually before I was 18 - I'd never had any experience with computers. Maybe as calculators? We had computer languages at school, it was called Russian Computer Language. It was all on paper - I was doing some updates, but that was all on paper.

I immediately started to work with computers. I started learning Fortran, assembly language. It was computer sigatone [?], they were clones of PDP-11. When I say I used to work with PDP-11, people think I'm very old. But it is not, because our clones were much younger than original PDP-11 computers.

So yeah, it was also assembly language, and I got really interested in programming. One of my first achievements was, as a compiler, I translated by hand, 800 lines of a Fortran program into 2000 lines of assembly language, and got a 25% increase in speed. But that was not that much. The program was actually calculating fuel for Russian space agency. At that time, if you remember, there was Buran).

Len: Yeah.

Dmitry: The first shuttles in the 80s. The program calculated for weeks some properties of rocket fuel. So, a 25% increase was very great.

So,o I got started with chemistry, but my heart was in computers. In C, I started doing a lot of programming in our chemical department. A lot of experience.

But then - by accident, again - there was the breakdown of Soviet Union. Life was not very good for being a student. It was beginning of 90s. And by a stroke of accident or luck, whatever - I joined a company from the United States that was doing speech recognition. It was called Covox. It was one of pioneers in computer [speech] recognition, and they were competitors of Sound Blaster, if you remember this company?

I spent like seven years there, doing some stuff. After that, I joined another company, also working from Moscow. It was related to image processing, Accusoft - very famous in States. But that was all work, I would say, from home, or from some remote office.

Just before coming to Ireland - in 2001, in January - I worked for a couple of years for a very big system integrator in Russia, where I also learned modern and software engineering practices. twenty years ago in January in 2001, I arrived to Ireland, and started working for Ericsson, and some other companies. I spent 14 years at Citrix, which I really liked.

Actually, regarding my career - I gave a lot of thought about it before our interview. I found out that I probably have three distinct careers.

One career is software engineering. That started, I would say - in 1993, probably? Or even earlier, if you count me programming in the chemistry department, right?

The second career is software technical support. It started when I joined Citrix in 2003. It was actually going in parallel with software engineering. But fourteen years in technical support is very good experience - because most software engineers, they don't have experience with software technical support, working with customers and understanding their mentality.

Third of course, I would say, is career is in writing and publishing.

Len: Thanks very much for that summary. I'm sure it was hard to keep it so compact. I really like that you had sort of three phases in the end.

There's a lot to talk about there. One thing - you mentioned perestroika. I think that actually, probably about half of our audience doesn't know what that is. So, I was wondering if you could talk a little bit about what life was like for you in your early days as a student, and what perestroika was?

Dmitry: Okay. So perestroika, it was a complete breakdown of the Soviet, I would say its late-socialism style in Soviet Union. It was changed, and there were glasnost also. Glasnost means you could talk about different topics freely.

So, I would say life in that time gradually become - free speech was everywhere, probably at the end of the 80s, beginning of 90s. In some sense, free speech was even more free than in other democratic countries of the world at that time. So, even in Russia. There was Yeltsin, one of the first Russian Presidents.

Life was very good initially, during the initial stages of perestroika, it was very good. But then all this breakdown was associated with changing from socialism to capitalism. It was in some sense abrupt, the coming of wild capitalism to Russia. So the life for students was not very good, if they were not backed up by employment.

I remember one day the prices were like five times more expensive. Imagine for students - it would be very hard. I did some rough times for a year or something like that.

But then I say I was very lucky, because I joined - working for American company. Imagine, my first salary was just $50, and it was enough to survive. I mean, like life was very luxurious. Life seemed very fast-growing, yeah. The salary was good.

Len: Was perestroika, did that mean "the opening up," is that what it means?

Dmitry: Yes, opening up, also to the whole world, but it's also opening up internally.

You could speak a lot of things, you could read a lot of books, and it was a very great experience. Also, computers were coming to Russia and the Soviet Union. So it - like the end of the 80s, beginning of the 90s - there were personal computers coming to Russia. So it's a very, very interesting period.

Len: That's really fascinating. I've spoken to people on this podcast from all around the world about their first experience with technology and things like that, but it - your experience of being sort of in this very serious, I mean, multiple phases of transitions.

Dmitry: Yes, I started with serious computers.

Len: And when you mentioned the sort of - the Olympics and stuff like that, for chemistry and things like that - I think one thing that it sort of might be difficult to communicate to people who aren't aware of the history, is that this was a very important thing for the Soviet Union, to compete globally on numerous things, right? The idea that you would be recruiting students to participate in this - there was a self-aware idea that we're sort of building the reputation and trying to sell, send the message that this is a good way to live in our country.

Dmitry: Yes.

Len: Then all of a sudden, it's like actually perestroika comes along - and it's like, "No, that's not the right way to live." It must have been a very -

Dmitry: Yes, just a complete breakdown of mentality, yeah, of many people.

Len: So you started working for this American company in the early 90s?

Dmitry: Yes.

Len: If I get the timeline correctly? You said you were working from home, so how did that work?

Dmitry: Oh, I would say, a lot companies - okay, the opening of society, also resulted - there were a lot of entrepreneurs coming to Russia, to Moscow, St. Petersburg or whatever, other cities at that time - actually to buy resources, like people resources. There were a lot of scientists working in Russia, like for example speech recognition or speech things. It's - everything was related.

Len: Or rocket fuel.

Dmitry: Yeah. I wasn't a scientist in speech recognition, but I was very good at system programming. Also, Windows at that time, started to develop Windows 3.1, and all this. So it was great time. So Microsoft -

Len: That's fascinating. So were you working online? You had the internet and the World Wide Web, was that a sort of popularly -?

Dmitry: Oh no, no. Initially we used the internet, I remember it was coming something about 1996 to Moscow around that, if I remember correctly. But before they used BBS - bulletin board systems, email, all this thing.

Email - not that probably like internet type - but something different. People were coming also - oh, people were coming just to Moscow to visit like once in a month or two months.

I worked from home, I worked in my laboratory, also in Moscow State University. Then from home as well, and later we had an office. So it was quite interesting.

Len: That's just so fascinating. I'm sure you've got a wealth of stories to tell.

I wanted to go into a little bit of detail about that specific example you gave when you were in school, I believe - of translating code by hand from one language to another. That's just so interesting. I'm very curious about the process. So you were given something probably on paper, and told to translate it. You would have had paper books on Fortran or whatever it was -

Dmitry: Yes.

Len: Then you would have had to submit your result. Did someone else then enter that code into a machine and then run it?

Dmitry: I was actually - the code was already there in Fortran and I was actually writing it, I think I was writing it also in some editor.

Len: Okay.

Dmitry: But at that time, editors were very, I would say, simple. They're not that powerful, like now. I was just translating line by line. Assembly language was not very, I would say, complex for PDP level, but mostly it was of course floating-point style. And it was line-by-line translating. Even translating print statements as well in Fortran, and then running that to see if you get the same output.

Len: As you mentioned, I guess you've had a couple of phases of your career, and a couple of phases of your life where you were living as well. You moved to Ireland in 2001, I believe - is that correct?

Dmitry: Yes, in January 2001.

Len: I'm just curious, I mean, I myself have moved from one country to another to live and work and things like that - and so have many other guests on the podcast. I'm curious about that. Was it easy for you to move? I mean, in a legal and citizenship sense, was it easy for you to move to Ireland from Russia?

Dmitry: Oh, yes. I can talk about this freely, it's a wonderful experience.

Len: Okay.

Dmitry: First of all, you can't believe I actually - I was actually living and working in Moscow. But I was recruited in Ukraine. Because Moscow - the life in Moscow was good, and not many people were going to - I would say, were willing to go to Ireland at that time.

Len: Okay.

Dmitry: But from Ukraine - a lot. So the recruitment was in Ukraine. I flew from Moscow to Kiev, had some interviews there, was hired, flew back to Moscow, then I applied for my working visa.

Actually - just shortly before that, Ireland introduced the so-called working visas. So you could freely come to Ireland, work up for two years - then renew your working visa - but you were not dependent on your employer.

If something was coming, like a lay-off or you wanted to change employment, then you could freely change your employment - go to another employer. It's different in other countries. If you lose your employment, you have to come back. So that was one major, I would say, good point for me, to come to Ireland.

I came to Ireland actually to learn English. I'm not - I may not be very well in speaking, but in writing or reading, I'm very good. So, I mostly concentrated on reading and writing. You have to to put something aside, you can't learn all these things.

Len: Well, I think you're speaking very well, and of course, your writing is very good as well.

I imagine that was probably around the time when Ireland was sort of becoming the - what they call the Celtic Tiger at the time.

Dmitry: Yes.

Len: They were competing very aggressively to become a place where people, particularly, I imagine in technology and finance and things like that, would work. I didn't know that detail though, that's very interesting that they were being open about sort of immigration and work status, to encourage people to come and stay, and not be afraid of losing their job - but actually be able to find better ones.

Then you've got this pool of local people who are already there, that the companies that are growing can draw upon - without having to worry so much about the sort of minutiae of exactly transitioning from one job to another, and things like that. That's a really fascinating detail.

I'm curious, how did you like it when you -? I imagine you've probably been living in Dublin the whole time, but was it a big shock?

Dmitry: Yes, it was very big shock. Actually, I arrived to a small town in the middle, at the heart of Ireland, called Athlone. Not to confused with CPUs that were popular at that time. Intel 1s or [?], I don't remember.

It was long time ago, but there was a popular joke about - so, Athlone was a great small town, but imagine you live in a very big city with millions of people, and then you arrive in a small town, and then you walk twice as fast as other people initially. But I liked it, and after some time, I found a new company. I was persuaded by a recruiter. So life in Dublin is better. I mean, in the sense you could see more, like in big cities. So shortly, in less than a year, I moved to Dublin. Since then of course, I live in Dublin, and I really like it.

Len: I imagine people must have been very interested in you in that small town as well. All of a sudden here's this Muscovite amongst us.

Dmitry: Oh, yes. People actually - Irish people are very friendly, it was a great experience.

Len: So professionally then, you ended up working for Citrix, I believe? Was that while you were in Ireland, that you started working for them?

Dmitry: No, initially I started working for Ericsson.

Len: Oh, for Ericsson, sorry.

Dmitry: Ericsson, very nice, and they still have very nice office. And then I actually moved to another company, called Programming Research, it was compiling frontends for C and C++. It was a very good experience, learning about the semantics of programming languages, C++.

There, I finally learned C++ well and - but then after, I would say - one year and half probably - I moved to another company. I was a contractor, consultant - it was my first security company.

Then after doing some consulting work there, contracting, I finally settled in Citrix, and Citrix was a really great experience for me. I would say - at that time, my career forked into software engineering and software technical support. I also started visiting the United States for work purposes, so it was also a great experience. I was in Florida, I was in Seattle. I really liked that - the whole fourteen-year experience, was really good for me.

Len: You mentioned the shift to working with customers. Was this like big corporate customers that you were working with, when you were doing support?

Dmitry: Oh, yeah. Citrix is a very well-known customer company - providing remote desktop services, external services, Windows, and also observation of internet traffic. A very well-known company, and they have cloud services now. The customers were mostly EMEA, from Europe, the Middle East, and Africa, the UK. It was all customers, I would say - from Fortune 500 companies, to small companies. But mostly my work was about internal customers.

Len: Oh, I see.

Dmitry: But from time to time, I was also engaging with outside customers, and this was really good experience, to see everything like that, and so -

Len: I'm just looking at your profile on LinkedIn, and this will come up when we start talking about your books and things like that, but you worked on - you performed diagnostics, and you did anomaly detection and things like that. I was just particularly - you say here, or it says on your bio, "Analysis of software execution artefacts such as memory dumps, traces and logs."

Dmitry: Oh, yes.

Len: I just wanted to take the opportunity here to link what we're going to be talking about with your books, to your career. For anyone listening who might not know anything, let's say they don't know anything about this stuff - what are memory dumps?

Dmitry: Okay, it's basically - I had a management training at some point, when I was a manager. The guy also asked me this question, and I answered. Then he replied, "Oh, I understand. That's computer psychology, like psychoanalysis or whatever, of computers." But of course, I won't provide this answer.

Basically, you have memory in your computer, and if something happens, some wrong behavior, like for example, the computer stops working, or hangs, or is frozen - you just take the contents of that memory from your computer, put it into some file, and then you send it to some analyst to look at it.

Then you look at this memory, and try to find who was responsible. Maybe this is an operating system vendor, or maybe that is some other company - several companies? Because sometimes there are conflicts.

Then you analyze, and you work in a sense like you are a general practitioner or GP, like in medicine, providing diagnostics. Then if you have enough knowledge, you may even fix it.

But most of the time, you won't be able to fix it, because, say the problem lies in the operating system vendor, and then you advise, "Go to that specialist." It may be Citrix, or that may be Microsoft, or that may be Linux, Unix. So, that's what I mean when I say, "diagnostics."

Initially, I was calling on that memory dump analysis. But then gradually, I realized that it's actually software diagnostics that I was doing.

You also have software logs; this is similar to narrative in medicine. You provide stories. Programs run, they execute, they do something - and then they write their own stories. They write their own stories in a log.

I actually found the so-called software narratology, because I was actually thinking about how to analyze all these logs. Then I found out that narratology, stories - is actually a close approximation what computers were doing, so -

Len: I'd definitely like to talk about that. This is so fascinating.

Just to start at the very detailed level, because this connection between psychology or psychoanalysis, the stories that we tell - and then what computers do and what we do with them, and how we try and understand what they've done - to talk about that, we need to understand a little bit, just at a very basic, non-technical level about how a computer works.

I remember a friend of mine, who - I've never studied Computer Science. I'm Leanpub's resident non-technical person, although I'm pretty technical in a lot of ways. But the way someone once described it to me was like, "Okay. A computer is like a list of tasks. Imagine it like a big stack of paper, and the topmost task is the first one that you do, and then you tick that one off. Then you do the next task, and the next task - and it's just that it does it incredibly fast."

Dmitry: Exactly.

Len: "There's this stack of tasks, and they go in order." But what you're saying here is that the computer will have a memory of what it did.

Dmitry: A brain, literally, yeah.

Len: Yeah. Exactly and much better than our own memories.

Dmitry: Yeah, and that talks very fast. They talk really fast. They talk so fast, that they are narrative - maybe like gigabytes, or terabytes, of information.

Then you have to look at that narrative, and try to find out what actually computer was saying.

Len: It's so fascinating, to talk about what the computer was saying, and to think about in those terms. Because I think for a lot of us, us ordinary computer users - if something breaks, we do the equivalent, sometimes literally and sometimes figuratively - of just unplugging it, and plugging it back in. If it works again, we're fine, we just carry on.

But if it's someone like you, or something very serious is happening, and then you actually need to look into it and figure out, maybe we got it working again by banging the machine or something like that? You have to understand what actually went wrong.

I think that if you're not familiar with how this technology works, you might think, "Oh, you'll just go in the memory task by task, and then you'll find the one that failed and that'll explain it."

Dmitry: Exactly.

Len: But obviously it's far more complex than that, and you're not always going to find the answer. But what you particularly, I think, were interested in, and are interested in - is these patterns, right? You've got this encyclopedia of over 300 patterns that you've identified over the years. I was wondering if you could talk a little bit more about that connection to narratology.

Because the computer's telling you a story, the memory's telling you a story. But then you've got to look for that story, and you've got to find what those patterns are. Then you've got to communicate them to someone to fix the problem or explain what it is.

Maybe if you could talk a little bit more about narratology, and how this all fits in? Maybe even if you can think of a specific example, where you were working on a particular project, and you were given this challenge to figure something out - how you would use this in your professional life, to deal with what you've got to deal with?

Dmitry: Okay, that's a great question, and about patterns. The patterns were very popular before I actually started working in technical support. But they were mostly used in designing and implementing or architecting software.

This phase, I usually call like "construction phase." But apart from construction, once software is constructed, you actually ship it to customers, and it could be running in a big data center, spanning 10,000 computers. The company may have 10,000 desktops. Then, something may happen, and you have to analyze, what's the problem, or find the route cause, or do some diagnostics.

This phase, I usually call "post-construction." Here you also need to have patterns. We have a problem. We need to have some vocabulary or language to describe these problems, and solutions to them.

Basically what I came upon, was the idea of analysis patterns for post-construction. We have problem, we have to analyze it. Then you have common ways, common techniques to do that analysis, and find something.

These analysis patterns, of course they are completely independent from just patterns. What is just a pattern, could be your computer crashed, in a particular module.

Or a simple example is - now you have printer problems, came to spotlight security. But in the past, printer problems were abundant. Because printer drivers were designed for home use. They were not designed to use in corporate environments where you had like say 100 users suddenly starting to print. And then the printer driver would misbehave and crash the print spooler - the printer, print manager. Then the print manager would actually dump itself, or create a system which would dump it into a memory dump.

And that memory dump would end up, like say - it would be sent to me. I would look at it and find out, "Okay, this particular print driver vendor is at fault." But this is a pattern, right? How I would find out that this particular driver was at fault? This is an analysis pattern.

Basically, this encyclopedia talks about analysis patterns. There could be some patterns there as well, but mostly it is analysis patterns. Because otherwise, there could be millions of patterns.

Len: I think like the - I mean, it's even stronger than an analogy with what happens in psychiatry. For example, what's the big manual called? The [DSM](https://en.wikipedia.org/wiki/Diagnostic_and_Statistical_Manual_of_Mental_Disorders or something like that?

Dmitry: Yes, exactly. Yeah.

Len: I'm not going to try and derail this too much, but that's actually a real interest of mine. is that one of the things that people - people misunderstand a lot of what happens in psychiatry, because psychiatry made the terrible unscientific mistake of using ordinary English language words to name the patterns that it - and there's always this little bit of a dialectic - are you identifying the pattern, or are you creating the pattern? It's both at the same time.

People would understand what psychiatry and what psychologists or therapists are up to much better, if instead of using words like "depression," they named it, like, "Pattern X1592."

Then you could just say, "Oh, it's pattern X1592." You go look that up, and you see there's some paper where someone defined pattern X1592. It's, "Under certain conditions, we can observe behavior XYZ and response to stimuli 123."

Then you'd be, "Oh, okay, that's -" But what I'm talking about when I'm talking about - quote unquote - sort of, "depression," is actually an identified pattern, a pattern that can be identified according to various acceptable methods. What you're talking about is understanding the behavior, and in particular - the interesting thing is, we often get interested when things go wrong.

Dmitry: Oh, yes.

Len: So we want to be able to - I mean, now, when you start talking about fixing, it gets more problematic, when you're talking about people's minds, rather than computers. But still, the idea is very similar to what's going on.

Then, when you talk about having a shared language, well, why do we give names to these patterns and then have manuals or encyclopedias? It's so that we can talk about them. Once we can talk about them, then we can resolve or address the situations.

But what's happening in psychiatry, and what's happening in what you're talking about, are not just analogously similar, but are actually substantively similar.

Dmitry: Yes, and I would say psychopathology is also one of my favorite - I have a few books on that. I'm not that deep into it, but I'm trying actually to learn from various disciplines. Because especially -

Len: Yeah, you talk about software pathology, right?

Dmitry: Yes, software pathology is, I would say - more related to the facts, and similar to pathology, because pathology is now used everywhere.

For example, people in data science talk about data pathology. So why not to talk about software pathology as well? Especially is pathology has a log suffix that actually - as a joke, maybe related to logs - but it's not. But starting, talking about patterns, analysis patterns, I also apply the same approach to analyzing logs and traces. How you would - you would analyze a trace or a log? Not just finding a particular web server crashed at a particular point in time, that would be concrete pattern; or a general pattern, like web server crashes - right, in a particular module. But how to analyze them, what -?

Len: What are traces?

Dmitry: Okay, what are traces? Traces, there are different, I would say, definitions of that. I usually use traces and logs interchangeably. For me, it is the same thing. But nowadays people talk about trace in context of distributed tracing.

I was lucky enough, actually, because all my trace analysis experience was distributed. I didn't know it was - it was only recently called "distributed tracing." And just simple, when a program outputs some statements on the screen, this is usually nowadays called "logs."

But I use them in the same way. I mean, it's just - I would say - a marketing description. Some people, if I say "trace analysis patterns," people would say, "Okay, is it related to distributed tracing?" But if I say "log analysis patterns," people would think, "Okay, it's about logs but not traces." I made up these trace and log analysis patterns recently.

Len: Is tracing, for example, I mean - what is a trace, in the context, say, of cloud computing, or something like that?

Dmitry: Okay, so in cloud computing, you have a lot of services - native services, native applications, you have operating system calls. Everything is interconnected. You have containers, you have container management, the components - everything. Then they all talk, they all output their logs, and then you want actually to correlate them.

Your small service, say, crashes. But you want to see what happened at that time in particular, what broke or operating system - what's going on there. You want to combine as much as possible, in some intelligent way.

Of course you can't combine everything, it's too much. This is an idea behind distributed tracing - to combine individual small logs, so you could do some correlation - similar to data science, data analysis.

Len: Okay, so it's a cross - I had a crude idea of tracing being across different specific machines. But you're saying it's the different programs that are talking to each other as well, and they'll each have their own log and then you -

Dmitry: It's not even talking to each other, we also, the program - like I would say - say you have a debate in Congress or in parliament, right? You have twenty people talking at the same time, and you actually - now someone, they may be talking to each other - then someone starts shouting and then ranting. You have transcriptions of everything. You want to correlate - okay, that guy said something about taxes here. Then another one at the same time was silenced, silencing himself. Herself.

Len: Oh, that's just super -

Dmitry: Okay, that silence may be a sign, actually, why that deputy was not speaking for this particular topic. The silence itself would be also some sign. You have to do correlation between all different, I would say - narratives, software narratives.

Len: That's incredibly fascinating to me. I mean, particularly the idea of doing a analysis of simultaneous speakers and what they're saying. Because - my academic background is in English literature and in philosophy.

Dmitry: Right, oh.

Len: And of course the model that we have in our - typically in philosophy for analyzing a debate - is one speaker speaks, and then another speaker speaks, and then another speaker speaks, and then another speaker speaks.

Or even when we're analyzing different works of literature in relationship to each other - it's always there's the whole coherent thing, and then there's the next whole coherent thing. But the idea of doing some analysis of multiple speakers at the same time, and then understanding the meaning and the interplay of those as interacting forces - and what the outcome is, for example - is just really interesting. I'll have to go away and think about that. But on that note -

Dmitry: Sorry I interrupted you. Actually you mentioned English Literature studies, and I was particularly interested in narratology and critical theory as well, so I read a couple of books. You want some trace and log analysis pattern names, you are influenced by critical theory and narratology.

Len: Oh, can you think of a couple off the top of your head? That's really fascinating.

Dmitry: For example, defamiliarizing a fact, something that you read.

For example, in particular - I would say, a book - then something that triggers you, you see something as completely unfamiliar. This also happens in trace and log analysis as well, and the other - a couple of others, I would say.

I published a book called Trace Log Analysis and it's now I think in the fourth edition? But I change the title, so the latest title is called Trace, Log, Text, Narrative, because I finally realized that the same ideas could be applied also to text.

Because you can actually break text into similar - like you have traces and logs, and apply the whole apparatus of a trace and log analysis. There are more than 200 patterns there.

Len: That's really fascinating, particularly the concept of defamiliarization, for example. For those who might not know, I'll try - I mean I haven't talked about this in years - but to try and explain it to people a little bit. It would be like - imagine something that you take for granted, suddenly becomes very present for you. An example of that might be the sight - the sudden presence in your environment of someone wearing industrial-level hearing protection and operating a leaf blower. To a lot of people, that's just a part of the landscape, every once in a while. But sometimes you can have this moment where you blink and go, "What on earth is someone doing on a street where people live, operating industrial machinery that's so loud it damages hearing?"

When you start thinking about things - ordinarily you just think, "Oh, there's someone doing yard work." But when you start bringing these other categories of explanation to it, then it can become very strange to you all of a sudden. Something that was so familiar, you didn't even notice it, when you come at it from another direction, you suddenly see it in a new light.

So the term, "defamiliarization," is partly about that thing. That can actually be part of like if there were sort of - that's when things start getting very political as well. Because that's where it would be - in Marxist theory, it would be trying to make the structures of capitalism defamiliarized.

It's like, "Why is it that behind that window is a loaf of bread, and I'm hungry, but I can't have it unless I've got some specific bills in my pocket that I can give to the person on the other side, who's protecting that loaf of bread?" Things like that, that's so interesting.

Actually, just on that note - and this might be a good opportunity to segue to - you said there was this third part of your career where you got into publishing and writing, and you founded something. I'm just looking up the name of it here, I had it here OpenTask Publishing, which I believe you founded in 2008, and I'm hoping that was a good opportunity for you to talk about that shift in your career.

Dmitry: Yes.

Len: So how did that come about?

Dmitry: Okay, yeah, it's an interesting story actually. I was an avid reader of many books in software engineering, so I always wanted to write my own book. But it's one thing - many people want to write books, and many people don't start - or start, and then don't finish.

I wanted to write a book about, as you guessed - speech sound recognition, because that was my first interesting project.

But what happened actually was, due to some luck, I started blogging, I think in 2006? I started blogging, and immediately I got that idea - as a software engineer working in technical support, I got that idea of patterns - analysis patterns. I started writing about crash dump analysis patterns, because this new field was completely unexplored before, so there were actually a lot of things to write.

Many people start blogging, but they don't have much to say. Not because they don't have the ability, but because the field is already taken, you can't just say original things all the time, or don't have ideas.

But here I had plenty of ideas. Software was crashing every day, so like lots of things - and with the help of debuggers, I could also write additional text. Because you could include snippets of debuggers, debugging output - and then it gives you text -

Len: Right, specific examples, yeah.

Dmitry: Yeah, specific examples. And then roughly, in 2007, I believe? There was one book published called The Old New Thing, about Windows, API. It was actually a collection of blog entries. It was published by a big publisher, and I thought, "Okay, maybe I should actually publish my own blog as a book."

I was thinking about these lines, and also at this time, there were some sites like Lulu, for example. You could actually upload your PDF file, and immediately after a couple of days, you could get your book printed.

I started with printing out a debugger output, so I got pretty heavy books called Reference Tech Traces. It was a very good experience, I would say - exciting experience, to upload your PDF file, and then suddenly have your book - it's similar like to DIY projects with electronics, or something that you can hold in your hands.

And then after that, I decided, "Let's try it with this blog." Some people actually approached me when I announced that I would be publishing. So, "What publisher would be publishing?". All the names were mentioned, like the big publishers. But I said, "No, no. Nobody contacted me from big publishers."

I decided to publish myself, so I had to learn. It was a very steep learning curve.

Initially I published on Lulu in hardback and paperback. But then I decided I actually wanted to have all this on Amazon. I started buying and reading books on how to publish on Amazon. There was no Kindle at that time, or CreateSpace, whatever. The only thing to do publishing on Amazon was to join Lightning Source. That was print-on-demand, that's now part of Ingram.

I got this account, and learned how to provide pre-media, create all these covers for books. I had to buy software, like Adobe Acrobat, to do all this, conversions, and all of what real publishers do.

Then I started publishing, and my books started appearing on Amazon. That was a great success, especially for the first volume of Memory Dump Analysis Anthology.. This is now in sixteen books, or fourteen volumes.

Then I started adding more books, and more books in color, in paperback, in hardcover. Lately, I also started selling in PDF, in EPUBs. I think in 2008 or 2010, O'Reilly Supply Books Online approached me.

I signed a ten-year contract with them. I also - for ten years, I had a very good channel for content publishing, and also books, 24 by 7. Also, as well as my published my first editions. It was a great experience.

But a good story always comes to end. Supply Books Online decided to stop cooperating with small publishers, and soon my ten-year-old contract finished. They actually said, "Okay, we are now concentrating on our own stuff, learning."

Then I was looking for another channel, and then I recalled that a year or so ago I learned about Leanpub. Then, I started putting all my PDF books, and EPUB variants where I had them, on Leanpub.

Initially, it was very slow, but actually it's a great success - because some books, I actually only published initially on Leanpub. It was Visual Category Theory, Lego stuff -

Len: I'd like to talk about that in a bit too. That's a really great series there.

Thank you very much for sharing that story. It's interesting, your trajectory with publishing is similar to Leanpub's in a way, and it's the same timeline almost. Leanpub started out as a blog to book platform and my -

Dmitry: I think probably maybe I heard about it, blog to book? I think would be -

Len: And actually my cofounder Peter's first self-published book was on Lulu.

Dmitry: Oh.

Len: If you want to read about that story, you can go to leanpub/lean and read all about it. His experience there - I won't go into it in detail but was - it's so wonderful to have an actual, physical copy of your book. But along the way, he was publishing it as a digital book-in-progress, and he would give people a code when they bought the book on Lulu, to go to a - basically a website, where they could comment and stuff like that, and give feedback.

The book that he ended up publishing in print on Lulu when it was done, had been built along the way - and that was how the idea for "Lean Publishing" came along.

But as you're talking about, back in the day, getting a print book was really complicated, everybody felt like they were reinventing the wheel every time they did it. There is still, to some extent, an element of that. But it wouldn't have been without banging our heads against the same wall stubbornly over time -

Dmitry: I say that - sorry, Len. I say that I have this publishing career, this separate career. Because I had to learn so many things. Once, I was even thinking, let's write a publishing series - just where I only put things relating to publishing, and completely omit computers. Like all different things. I mean, at one point - produce a publishing series.

Len: Oh, I think that would be a really interesting story to tell. Because I think, and I think there would be a lot of other people, who would be fascinated to hear about someone who's - there are a lot of other people who've gone on the same path, and I think a lot of people feel very alone in that.

But actually - there's many authors all around the world who've struggled up against the same things, and succeeded and failed in the same ways. and things like that. It's a really fascinating story.

This gives us the opportunity - towards the end of the interview, we get very much into the weeds of writing and self-publishing, and things like that. I wanted to ask you how are you - how do you write your books today, and how do you produce them? What apps do you use, or anything like that?

Dmitry: I am pretty much a Windows user, because most of my life experience was in Windows, not counting MS-DOS. But I would say, yes - I use Microsoft Word.

Len: Okay.

Dmitry: Stream to PDF later on, and then use Adobe Acrobat to do all finishing. I use Adobe Photoshop for image conversion. Because sometimes you need specific images not in a color space, but also with some amount of grayscale - otherwise you won't be accepted by a real production environment.

Len: Oh, that's fascinating. You write in Word, and then you'll produce a PDF - and then you'll add the images in Adobe Acrobat, is that how it would work? Or would you put the images in in Word?

Dmitry: No, like basically it forks into online. My pipeline is now very organized now - when I do everything in Microsoft Word - but then I fork into print, into online.

Len: Oh, I see. Okay.

Dmitry: Online have all the images added - are PNG or JPEG, are all fine. But for print, especially now, I do colour books mostly, so I need to add the same pictures, but in a different color space - probably a different property. For example, less black grain. All these adjustments, I need to do in some software.

But at some point in time, a long time ago, I bought Adobe Acrobat. I tried with open source, and I couldn't do quite the same thing that commercial Adobe Acrobat was doing. I had to buy Adobe Acrobat. Expensive, but actually it was worth it. For ten years, I am using the same Acrobat, and then I put some file for publishing, and online I distribute on Leanpub or sell separately, or - but if I want to do an EPUB, I usually use iAuthor on Mac.

Len: iAuthor, okay.

Dmitry: On Mac Pro, some computer. That's an excellent tool to produce EPUBs, you just load a Microsoft Word file, and then you do some corrections - it's very straightforward now. Or iBooks Author or something? I don't remember correctly how it is called.

Len: That's really great to hear those details, especially the forking from ebook to print. Dealing with images and things like that, is something that can be really difficult. But once you - as you say, once you've got your pipeline organized, and that can take a long time - it can be a really streamlined process. And, I - just a very particular point to make for anyone listening, Leanpub has a Bring Your Own Book writing mode, which is what Dmitry uses.

Dmitry: Yes.

Len: You can use all of our awesome - I mean, well I'm speaking proudly of our own product. But you can use all of our awesome features for in-progress publishing, and all that stuff, without having to write in a Leanpub writing mode process. You can actually have your own totally independent process for producing PDF, EPUB and MOBI, and you can still publish your book on Leanpub - and that's what Dmitry does.

Dmitry: Yes, that's great. One feature I like on Leanpub as well, is - sometimes people return books, and for different reasons. I like that they are able to write some comments why they actually return it. Sometimes, it triggered me writing free books.

For example, one user of Visual Category Theory Brick by Brick complained that he has difficulty understanding the mathematical language used. Then, I immediately produced another, Part 0 - a free one, also published to Leanpub. It's free, it actually explains this language in some detail.

For me, it was kind of - one story was - I made a mistake in one of the books, like the repository for the source code for projects wasn't correct. The customer actually returned the book because of that, and I immediately corrected and kind of -

Len: Thanks very much for sharing that story. That's something we don't talk about a lot here. Well, we do talk about refunds sometimes, but it's - people who might not know, on Leanpub we have a what we call our "100% Happiness Guarantee," which is that within 45 days of any purchase, you can just do a couple of clicks and you can do a refund. But for anyone who - I mean, for Leanpub authors listening who don't - who maybe have never asked for a refund themselves before, if you request a refund on Leanpub, you can just click a button, request a refund, and then click another button, and it's done.

But we also have a little field where you can give feedback if you want to, that the author can see and they - by the way, they won't see who you are - they won't get any personal information about you when you do that. If you haven't done it yourself, Dmitry - we actually have what we hope people take in good humor, a passive-aggressive little paragraph that says, "We're sorry you don't like this book for whatever reason, that's perfectly fine. This is how much money the author's going to lose if you click the button below. If you'd like to leave -" It's better said than that, but, "If you'd like to leave a message, you can do that."

Actually, this is a way that people have of communicating to an author, what they didn't like about the book. That can really help you, not only with the content of your book, or any mistakes or lack of clarity or things you could just do better - but also with the presentation of your book.

For example, people might say, "Oh, I thought this was a book about subject X, but then I found out it was about subject Y." Then you can use that to change your "About the Book" description on Leanpub - or you can change your subtitle, or even your title, if your book hasn't been out for very long.

That feedback, people often think, "Oh no, a refund, I'm losing X amount of money. Or someone really doesn't like me." It's often just they're like, "No, it's not that they dislike you, or they think it was a terrible book or something like that," and that feedback can actually be really helpful.

I don't want to let you go without giving you an opportunity to talk about your Visual Category Theory Brick by Brick books, which are really fun, and involve Lego. I was wondering if you could talk just a little bit about that series. You've got one for chemistry as well, I believe?

Dmitry: Oh, yes. It was a great addition to, I would say - it all started-- I mean, because I have kids, and they play with Lego all the time. Sometimes I was playing with them, and then I realized, "Okay, this combination of Lego blocks is something that can be represented from computer memory, or some data structure or something." Sometimes I even used a theme, like a block box. Put it on a panel and then represent traces and logs in such a way.

But at one point, I got thinking, I wanted to do something with my son about chemistry, some project for school. Then I got thinking about what actually to do.

I was reading some book about artificial chemistry. This is chemistry that could be symbolic with some, I would say - the atoms could be completely fictional. Like a universe could have 1,000 atoms, only three atoms. Chemical reactions could be completely made up, so that's great -

Len: Wow.

Dmitry: The name, called "artificial chemistry." It's used also in unconventional computing or studying life, and so many things. The idea came to me to use Lego to represent chemical structures. But not like in 3D. That was done before, but on the base plate. This naturally represents what we see chemical formulas, chemical structures on paper. It's actually very easy to reason, you have electrons, you can put small blocks - it's - put a reaction. It's very powerful.

From that, I came later to the idea to try mathematics as well, category theory. Because I wanted to write something about category theory some time before that, and had even started.

Because category theory, I learned by accident back in 2003, wasn't even connected to functional programming as it is now known - or I would say, Computer Science. It was just pure interest, and it was always associated with physics, mathematics, biology - something like that, category theory.

I never thought about it in context of programming. I knew it was Computer Science, but Computer Science of the past, not current functional programming languages - for example Haskell or Scala, and other languages.

I wanted to write a book. I started, but then I have actually a very long list - probably 200 books to write, probably they would never be written. And that was one of them.

I probably wrote two pages, and it was visual. I was using PowerPoint. I use Microsoft PowerPoint as well a lot. Visio, Microsoft Visio for diagrams. I did some diagrams and then I put it aside because - then, when I got this idea to use blocks, and after I actually was successful in depicting graphs using Lego - because I could use Lego for arrows between objects. And then I got this idea, "Okay, so we can depict category theory arrows using Lego as well."

hen I started all this series. I was thinking maybe one or two parts would be enough, and then I was able to do seven - and now have eight.

Tthen I got another idea. Some people complained that they don't like this Lego language for category theory. They like more traditional diagrams. I started another series called CoParts. Because in category theory, you have duality. You have objects, arrows between them - and then you have dual objects, dual categories, the same objects, but the arrows are completely reversed.

These coparts, the dual two parts - they have the same diagrams, but they are now traditional black and white. Plus there are also color diagrams, but they resemble Lego blocks. But they are normal diagrams, so -

I probably did two coparts, try and consume them actually - all this, to do it properly - and without mistakes. Because, to write the first, to write each part of category - before Category Theory I had to consult probably ten, fifteen books on category theory. Because it's very difficult to do it right, without mistakes.

Some authors in category theory, there are slight differences in explanations. I want to make it all coherent, so people won't complain. So far, no one's said anything like, "I found mistakes." Either people allow it, or they are completely happy. So, no one has pointed me to particular mistake in my Visual Category Theory books. "Okay, this is wrong."

Len: Well, I'm sure people do love it. For anyone who's been listening - there will be links to all of these things in the transcript, that you'll be able to find on the Leanpub website. That's just so fascinating. Congratulations on a book with one typo. That's a very big achievement.

The last question I always like to ask guests on the podcasts, if they're Leanpub authors, is - if there was any magical feature we could build for you, or any very annoying problem on Leanpub that we could fix for you - is there anything specific you can think of, that you would ask us to do?

Dmitry: Actually I will - yes - one feature is, I will think - I have many courses I publish into your format.

Len: Right.

Dmitry: Actually, I got a great idea - I can say it freely, I mean anyone can implement. I don't want to patent it. If you have a course in PDF - but in order to save time - you can publish a short course on Leanpub, for example. With questions and answers. I actually recently started writing quizzes for one of my books, and I love this. In the future, I will add more quizzes to my courses and books. But currently I have great courses. I only want to publish mostly quizzes, like questions, answers.

I want to provide a certificate. But I want greater customization for the certificate. For example, I tried with a course, but I wasn't able to make it very customizable. For example, I want to put my own image. So is it possible? What do you think?

Len: Okay, so you're talking very specific - just so I get it clear. For anyone listening, in addition to being able to use Leanpub to publish ebooks, you can also use it to publish courses.

Dmitry: Yes, exactly.

Len: Where you could, yeah - when you're creating a course, you can set up a certificate for someone to get when they complete the course.

And so, what you're asking for is more customization around the certificates?

Dmitry: Yes. So, for example, say Software Diagnostics Institute My name - include my signature and put - say - have a very nice background, for example, that would be more customizable. The course would be mostly questions and answers, using existing courses, because I don't have time to convert all my courses into a Markdown format.

I will use existing courses. But in order to get the certificate, you have to purchase this theme, course.

Len: Oh, that's really interesting.

Dmitry: Then you do some questions and answers. Like maybe 50 of them? It won't be easy. But, at the end, you get the certificate.

Len: In this process, how would a person get graded to prove -? Or do they get graded by someone else or some machine at all?

Dmitry: They already know this stuff, then they would answer correctly. Either they already know the stuff, like the subject. He answers correctly, he gets graded. Or he purchases, or she purchases the course. The person purchases the course separately - start this, then this is - I would say - course, or you may even, may get something differently, like - whatever is good.

It would be great - for example, for me - I'm not sure about other people. But for me, it would be great. Because I have - I would say, seven or eight - or even more courses.

Instead of converting them into Markdown formats, splitting into chapters, adding - I could devise my questions and answers for each say lesson. Then just pass - just do it like "Lesson one, ten questions." "Lesson two, ten questions." Or, "Exercise one," mostly I do in exercises. Or, I even have books like, Practical Foundations for Linux, or Windows, like debugging - they don't have exercises, but they have chapters.

Len: That's really fascinating. Thanks very much for that feedback. That's really interesting, and gives me a lot to think about.

Dmitry: I'll try this anyway - just to see how it works, maybe for one of the courses?

Len: That's really interesting. Thanks very much for that. Like I said, I have a lot to think about there.

Well, Dmitry, thank you very much for a really great conversation, and for taking the time out of a beautiful Dublin evening to talk to everyone about your career, and about your work, and about your books, and everything. It's really fascinating, and we could've talked a lot longer about a lot more things. But maybe we'll do another one in the future?

Dmitry: Yeah.

Len: And thanks very much for being a Leanpub author as well.

Dmitry: Thank you, thank you Len. That was really great.

Len: Thanks.

Dmitry: I really enjoyed it, great interview. Thank you very much.

Len: Thanks.

And as always, thanks to all of you for listening to this episode of the Frontmatter podcast. If you like what you heard, please rate and review it wherever you found it, and if you'd like to be a Leanpub author, please visit our website at leanpub.com.