Practical Foundations of Linux Debugging, Disassembling, Reversing


This book is no longer available for sale.

Practical Foundations of Linux Debugging, Disassembling, Reversing

Training Course

About the Book

This training course is a Linux version of the previous Practical Foundations of Windows Debugging, Disassembly, Reversing book. It also complements Accelerated Linux Core Dump Analysis training course.

Although the book skeleton is the same as its Windows predecessor, the content was revised entirely because of a different operating system, debugger (GDB), toolchain (GCC, assembler, linker), application binary interface, and even an assembly language flavor, AT&T.

The course is useful for:

  • Software support and escalation engineers, cloud security engineers, SRE, and DevSecOps
  • Software engineers coming from JVM background
  • Software testers
  • Engineers coming from non-Linux environments, for example, Windows or Mac OS X
  • Linux C/C++ software engineers without assembly language background
  • Security researchers without assembly language background
  • Beginners learning Linux software reverse engineering techniques

This book can also be used as an x64 assembly language and Linux debugging supplement for relevant undergraduate-level courses.

  • Share this book

  • Categories

    • C and C++
    • Computer Security
    • Computer Science
    • Computer Hardware
    • Embedded Systems
    • Testing
    • Cloud Computing
    • Operating System Development
    • Software Engineering
  • Feedback

    Email the Author(s)

About the Author

Dmitry Vostokov
Dmitry Vostokov

Dmitry Vostokov is an internationally recognized expert, speaker, educator, scientist, inventor, and author. He is the founder of pattern-oriented software diagnostics, forensics, and prognostics discipline (Systematic Software Diagnostics), and Software Diagnostics Institute. Vostokov has also authored more than 50 books on software diagnostics, anomaly detection and analysis, software and memory forensics, root cause analysis and problem solving, memory dump analysis, debugging, software trace and log analysis, reverse engineering and malware analysis. He has more than 25 years of experience in software architecture, design, development and maintenance in a variety of industries including leadership, technical and people management roles. Dmitry also founded Syndromatix,, BriteTrace, DiaThings, Logtellect, OpenTask Iterative and Incremental Publishing, Software Diagnostics Technology and Services (former Memory Dump Analysis Services), and Software Prognostics. In his spare time, he presents various topics on Debugging TV and explores Software Narratology, its further development as Narratology of Things and Diagnostics of Things (DoT), Software Pathology, and Quantum Software Diagnostics. His current areas of interest are theoretical software diagnostics and its mathematical and computer science foundations, application of formal logic, artificial intelligence, machine learning and data mining to diagnostics and anomaly detection, software diagnostics engineering and diagnostics-driven development, diagnostics workflow and interaction. Recent interest areas also include cloud native computing, security, automation, functional programming, applications of category theory to software diagnostics, development and big data, and diagnostics of artificial intelligence.

Table of Contents

Contents 4

Preface 9

About the Author 10

Chapter x64.1: Memory, Registers, and Simple Arithmetic 11

Memory and Registers inside an Idealized Computer 11

Memory and Registers inside Intel 64-bit PC 12

“Arithmetic” Project: Memory Layout and Registers 13

“Arithmetic” Project: A Computer Program 14

“Arithmetic” Project: Assigning Numbers to Memory Locations 15

Assigning Numbers to Registers 17

“Arithmetic” Project: Adding Numbers to Memory Cells 18

Incrementing/Decrementing Numbers in Memory and Registers 21

Multiplying Numbers 24

Chapter x64.2: Code Optimization 27

“Arithmetic” Project: C/C++ Program 27

Downloading GDB 28

GDB Disassembly Output – No Optimization 29

GDB Disassembly Output – Optimization 32

Chapter x64.3: Number Representations 33

Numbers and Their Representations 33

Decimal Representation (Base Ten) 34

Ternary Representation (Base Three) 35

Binary Representation (Base Two) 36

Hexadecimal Representation (Base Sixteen) 37

Why are Hexadecimals used? 38

Chapter x64.4: Pointers 41

A Definition 41

“Pointers” Project: Memory Layout and Registers 42

“Pointers” Project: Calculations 43

Using Pointers to Assign Numbers to Memory Cells 44

Adding Numbers Using Pointers 50

Incrementing Numbers Using Pointers 53

Multiplying Numbers Using Pointers 56

Chapter x64.5: Bytes, Words, Double, and Quad Words 61

Using Hexadecimal Numbers 61

Byte Granularity 62

Bit Granularity 63

Memory Layout 64

Chapter x64.6: Pointers to Memory 67

Pointers Revisited 67

Addressing Types 68

Registers Revisited 73

NULL Pointers 74

Invalid Pointers 75

Variables as Pointers 76

Pointer Initialization 77

Initialized and Uninitialized Data 78

More Pseudo Notation 79

“MemoryPointers” Project: Memory Layout 80

Chapter x64.7: Logical Instructions and RIP 89

Instruction Format 89

Logical Shift Instructions 90

Logical Operations 91

Zeroing Memory or Registers 92

Instruction Pointer 93

Code Section 95

Chapter x64.8: Reconstructing a Program with Pointers 97

Example of Disassembly Output: No Optimization 97

Reconstructing C/C++ Code: Part 1 99

Reconstructing C/C++ Code: Part 2 101

Reconstructing C/C++ Code: Part 3 103

Reconstructing C/C++ Code: C/C++ program 104

Example of Disassembly Output: Optimized Program 105

Chapter x64.9: Memory and Stacks 107

Stack: A Definition 107

Stack Implementation in Memory 108

Things to Remember 110

PUSH Instruction 111

POP instruction 112

Register Review 113

Application Memory Simplified 115

Stack Overflow 116

Jumps 117

Calls 119

Call Stack 121

Exploring Stack in GDB 123

Chapter x64.10: Frame Pointer and Local Variables 127

Stack Usage 127

Register Review 128

Addressing Array Elements 129

Stack Structure (No Function Parameters) 130

Function Prolog 131

Raw Stack (No Local Variables and Function Parameters) 132

Function Epilog 134

“Local Variables” Project 135

Disassembly of Optimized Executable 138

Chapter x64.11: Function Parameters 139

“FunctionParameters” Project 139

Stack Structure 140

Function Prolog and Epilog 142

Project Disassembled Code with Comments 144

Parameter Mismatch Problem 147

Chapter x64.12: More Instructions 149

CPU Flags Register 149

The Fast Way to Fill Memory 150

Testing for 0 152

TEST - Logical Compare 153

CMP – Compare Two Operands 154

TEST or CMP? 155

Conditional Jumps 156

The Structure of Registers 157

Function Return Value 158

Using Byte Registers 159

Chapter x64.13: Function Pointer Parameters 161

“FunctionPointerParameters” Project 161

Commented Disassembly 162

Chapter x64.14: Summary of Code Disassembly Patterns 169

Function Prolog / Epilog 169

LEA (Load Effective Address) 171

Passing Parameters 172

Accessing Parameters and Local Variables 173

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.

You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!

So, there's no reason not to click the Add to Cart button, is there?

See full terms...

80% Royalties. Earn $16 on a $20 book.

We pay 80% royalties. That's not a typo: you earn $16 on a $20 sale. If we sell 5000 non-refunded copies of your book or course for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earnedover $13 millionwriting, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub