Leanpub Header

Skip to main content
Go to Leanpub.com

Assembly Under the Hood: Modern x64 Programming for Windows

From Fundamentals to Advanced Techniques

This book is 100% completeLast updated on 2026-07-03
Steve T. Publications

Discover how Windows software really works with a practical guide to modern x64 assembly. From core concepts to advanced optimization, reverse engineering, and low-level systems programming, this book equips you with the knowledge to write fast, secure, and efficient code.

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
About

About

About the Book

This is a comprehensive guide to modern x64 assembly programming on Windows, designed for experienced C/C++ developers and motivated beginners who want to understand what happens at the lowest level of software execution. We cover everything from the x64 register file and calling convention through SIMD vectorization, PE format internals, structured exception handling, multithreading, reverse engineering, optimization techniques, debugging strategies, and secure coding practices. Each chapter builds on the last, with annotated code examples, practical projects, and exercises to reinforce your understanding. By the end, you will have a deep mental model of how x64 Windows programs execute and the skills to write high-performance, secure low-level code.

Share this book

Feedback

Email the Author

Bundle

Bundles that include this book

Author

About the Author

Steve T. Publications

Steve T. is a cybersecurity leader, researcher, and engineer with more than 20 years of experience across application security, infrastructure security, vulnerability management, software development, and secure engineering practices. Having built his career alongside the growth of the modern internet, he has worked through multiple generations of technology, evolving security threats, and changing development methodologies.

He is currently part of the advanced research organization at a leading cybersecurity company, where he focuses on emerging threats, security innovation, and the practical application of research. His work involves investigating new attack techniques, evaluating emerging technologies, conducting deep technical analysis, and helping organizations better understand and manage complex security risks.

In addition to his research responsibilities, Steve leads a team of senior engineers and subject matter experts who create technical books, training programs, and educational resources for security professionals. Through this work, he helps engineers, developers, architects, and security practitioners strengthen their skills and build more secure systems.

Steve's technical expertise spans software development, reverse engineering, web application security, penetration testing, security architecture, incident response, vulnerability research, operating system internals, and secure software development. His ability to analyze systems at both the source code and binary levels enables him to bridge the worlds of software engineering, security research, and practical defense.

Over the course of his career, Steve has worked with organizations across a wide range of industries, helping them identify, assess, and remediate security weaknesses in critical applications and infrastructure. He is recognized for combining deep technical expertise with a pragmatic approach to security, focusing on solutions that are effective, sustainable, and aligned with business goals.

Through his work in research, engineering, leadership, and education, Steve continues to contribute to the advancement of cybersecurity and the development of secure, resilient technology systems.

Contents

Table of Contents

From Fundamentals to Advanced Techniques

Introduction: Why Assembly Still Matters

Chapter 1: The x64 Landscape

  1. Why Assembly Still Matters in 2026
  2. The Journey from x86 to x64: AMD64 and Intel 64
  3. The Windows x64 Ecosystem: Toolchain Overview
  4. Development Environment Setup
  5. Your First x64 Program: Hello World in Assembly
  6. Exercises

Chapter 2: The x64 Architecture Deep Dive

  1. The Register File: General-Purpose, Flags, and Special Registers
  2. Floating-Point and SIMD Registers: XMM, YMM, ZMM
  3. Instruction Encoding and OpCodes
  4. Addressing Modes: Immediate, Register, Direct, and Indirect
  5. Data Types and Memory Layout: Bytes, Words, Dwords, Qwords
  6. The Condition Code Register and Flags
  7. Exercises

Chapter 3: The x64 Calling Convention

  1. Overview of Calling Conventions: Why They Matter
  2. The Windows x64 Fast Call Convention: Register Parameters
  3. Stack Alignment and the Red Zone
  4. Parameter Passing: Integer/Pointer vs. Floating-Point Registers
  5. Return Values and the RAX Register
  6. Caller-Saved vs. Callee-Saved Registers
  7. Variadic Functions and the Vector Register Mask
  8. Practical Examples: Calling and Being Called
  9. Exercises

Chapter 4: Stack Frames and Function Prologues/Epilogues

  1. The Call Stack: Structure and Purpose
  2. Frame Pointers vs. Frameless Functions
  3. Building a Stack Frame: The Prologue Pattern
  4. Allocating and Freeing Local Variables
  5. Aligning the Stack for SIMD Operations
  6. Exception Handling Frames (SEH) and the Frame Pointer Chain
  7. Debugging Stack Frames with Visual Studio
  8. Exercises

Chapter 5: Memory Management in x64 Assembly

  1. Virtual Memory and the x64 Address Space (8TB User, 256TB Kernel)
  2. Heap Allocation via Windows API (HeapAlloc, VirtualAlloc)
  3. Stack vs. Heap: When to Use Each
  4. Memory Alignment Requirements for SIMD and Performance
  5. Cache Lines and Locality: Avoiding False Sharing
  6. Memory-Mapped Files and Direct Access
  7. Security: DEP, ASLR, and Stack Canaries
  8. Exercises

Chapter 6: Working with the Windows API from Assembly

  1. Introduction to the Windows API and P/Invoke Concepts
  2. Loading DLLs and Resolving Function Addresses
  3. Calling Win32 Functions: MessageBox, CreateFile, ReadFile, WriteFile
  4. Unicode vs. ANSI: String Handling in Assembly
  5. Error Handling: GetLastError and HRESULT
  6. Practical Project: A File Copy Utility in Assembly
  7. Exercises

Chapter 7: SIMD Programming: SSE, AVX, and AVX-512

  1. SIMD Fundamentals: Why Parallelism at the Instruction Level
  2. SSE (128-bit): Registers, Instructions, and Data Types
  3. AVX (256-bit): Extending SIMD Width
  4. AVX-512 (512-bit): The Cutting Edge and Masking
  5. Data Alignment and Packing for SIMD Operations
  6. Practical Examples: Vector Addition, Dot Product, Image Processing
  7. Interfacing SIMD with the Calling Convention
  8. Exercises

Chapter 8: PE Internals: The Portable Executable Format

  1. The PE File Structure: DOS Header, PE Signature, COFF Header
  2. Section Table: .text, .data, .rdata, .bss, and More
  3. The Import Table: Resolving External Dependencies
  4. The Export Table: Sharing Code Across Modules
  5. Relocations and Patching: Making Code Position-Independent
  6. Debug Information: PDB Files and Symbol Tables
  7. Practical Exercise: Inspecting a PE File with Tools
  8. Exercises

Chapter 9: Exception Handling in Windows

  1. Exceptions vs. Errors: The Windows Model
  2. Structured Exception Handling (SEH): The Original Mechanism
  3. The Exception Registration Record (ERR) Chain
  4. Try/Except and Try/Finally in Assembly
  5. Access Violations and Debugging Crashes
  6. Modern C++ Exceptions: How They Map to SEH Internals
  7. Writing Robust Code with Proper Exception Handling
  8. Exercises

Chapter 10: Multithreading and Synchronization

  1. Threads in Windows: Creation, Lifecycle, and Threading Model
  2. Creating Threads from Assembly (CreateThread, _beginthreadex)
  3. Synchronization Primitives: Mutexes, Semaphores, Critical Sections
  4. Atomic Operations: Interlocked Functions and Lock-Free Programming
  5. Memory Ordering and the x64 Memory Model
  6. Thread-Local Storage (TLS)
  7. Practical Project: A Simple Parallel Counter
  8. Exercises

Chapter 11: Reverse Engineering Basics

  1. What is Reverse Engineering? Ethics and Legal Considerations
  2. Disassembly vs. Decompilation: Tools of the Trade
  3. Reading Disassembled Code: Patterns and Conventions
  4. Identifying Functions, Loops, and Data Structures
  5. Understanding Compiler Optimizations in Disassembly
  6. Practical Exercise: Reverse Engineering a Simple Program
  7. Exercises

Chapter 12: Optimization Techniques

  1. Profiling and Identifying Bottlenecks (VTune, Perf)
  2. Instruction-Level Parallelism and Out-of-Order Execution
  3. Branch Prediction and Avoiding Branches
  4. Loop Unrolling and Software Pipelining
  5. Register Allocation and Spilling
  6. SIMD Optimization Patterns
  7. Microbenchmarks: Measuring Real Performance
  8. Exercises

Chapter 13: Debugging Assembly Code

  1. The Debugger’s View: Registers, Memory, and Disassembly Panes
  2. Setting Breakpoints and Stepping Through Code
  3. Inspecting the Call Stack and Local Variables
  4. Data Breakpoints and Hardware Breakpoints
  5. Debugging Crashes: Access Violations and Stack Overflows
  6. Using Visual Studio Debugger for Assembly
  7. Command-Line Debugging with WinDbg
  8. Exercises

Chapter 14: Secure Coding in Assembly

  1. Common Attack Vectors: Buffer Overflows, ROP, JIT Spraying
  2. Stack Canaries and Canary Values
  3. Safe String Operations and Bounds Checking
  4. Control Flow Integrity (CFI)
  5. Secure Memory Management: Preventing Use-After-Free
  6. Information Disclosure and Side Channels
  7. Best Practices for Writing Secure Assembly
  8. Exercises

Chapter 15: Advanced Topics and Future Directions

  1. Kernel-Mode Programming: Introduction to Windows Drivers
  2. The Future of x64: Instruction Set Extensions and Trends
  3. Comparing x64 to ARM64: Cross-Platform Considerations
  4. Emerging Tools and Techniques in Low-Level Programming
  5. Resources for Continued Learning
  6. Exercises

Conclusion: The Assembly Mindset

  1. What You’ve Learned: The Big Picture
  2. When to Use Assembly vs. High-Level Languages
  3. Building a Mental Model of Low-Level Execution
  4. Resources for Further Study
  5. The Community: Forums, Conferences, and Open Source Projects
  6. Final Thoughts

References (Part 1)

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub