Leanpub Header

Skip to main content

Building Database Engines from Scratch

A Hands-On Guide to Storage, Concurrency, Query Processing, and Reliability in Systems Programming

Building Database Engines from Scratch
This book is 100% completeLast updated on 2026-08-30

What really happens between typing a SQL query and getting a result? Build a database engine from the ground up in Rust and find out. You’ll work through storage, indexing, concurrency, recovery and query processing while turning each chapter’s code into one working system.

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
211
Pages
About

About

About the Book

This book teaches you how modern database systems work by having you build one. Over fifteen chapters we will design and implement a complete relational database engine in Rust: storage layers that read and write pages to disk, a buffer pool manager that caches hot data in memory, write-ahead logging for crash recovery, B-tree indexes for fast lookups, a transaction system with proper isolation and concurrency control, a SQL parser, a cost-based query optimizer, and an execution engine that runs joins, sorts, and aggregations. Every chapter adds working code to the same project so that by the end you have a coherent, functional database application rather than a collection of isolated examples. If you are a software engineer who wants to understand what happens between typing SELECT * FROM users and seeing results on screen, this is the book for you.

Author

About the Author

Steve Publications

Steve is a technology professional with more than 20 years of experience in software development, server infrastructure, cybersecurity, vulnerability research and reverse engineering. Throughout his career, he has designed, secured, analyzed and tested complex software and infrastructure, with a particular focus on understanding how systems fail and how they can be made more secure.

Outside of work, Steve enjoys sharing knowledge with the technology community. He collaborates with researchers, industry experts and technology professionals to write practical books covering software development, cybersecurity, cloud computing, networking, DevOps, artificial intelligence and enterprise technologies. His books focus on practical learning through clear explanations, real-world examples and hands-on exercises. With more than two decades of industry experience, his goal is to help IT professionals, students and technology enthusiasts build useful skills and stay current in a rapidly changing industry.

We believe readers deserve to know how our books are created. Most of our authors are not native English speakers, so we use AI to help translate, proofread manuscripts, fix grammar, improve sentence structure and make technical explanations easier to read. AI is used as an editing tool only. It does not replace the research, technical knowledge or hands-on experience behind our books. Some of our authors also prefer to remain anonymous for privacy or professional reasons. In those cases, we publish their work under a different name. The author's name may be different, but the quality of the content and our review process remain the same.

Every book is written, reviewed and maintained by experienced technology professionals, with contributions from our private technical community of more than 420 engineers and researchers. We spend far more time validating technical accuracy and keeping our content up to date than generating text. We are always interested in working with experienced professionals who have deep expertise in a particular technology or domain. If you would like to publish a book with us or help review an existing manuscript, we'd love to hear from you. Send us a message describing your area of expertise. We are especially interested in niche technologies, specialized skills and emerging topics that are underrepresented in existing technical literature.

If you look through the contents of our books, you'll see practical examples, detailed explanations and material that is regularly updated. Our goal is to publish books that professionals can actually rely on, not low-effort AI-generated content. If you ever feel that one of our books does not meet that standard, Leanpub offers a 60-day money-back guarantee. Feel free to request a refund if you are not satisfied with your purchase.

Contents

Table of Contents

A Hands-On Guide to Storage, Concurrency, Query Processing, and Reliability in Systems Programming

Chapter 1: Why Build a Database?

  1. The Hidden Complexity of Every Database Query
  2. What You Will Build and Why It Matters
  3. A Tour of the Architecture We Are Going to Construct
  4. Implementation Roadmap: From Empty Project to Working Database
  5. How to Read This Book and Use the Code

Chapter 2: Project Setup and Systems Foundations

  1. Choosing Rust for Database Implementation
  2. Project Structure and Build Configuration
  3. Core Abstractions: Pages, Records, and Offsets
  4. Error Handling Strategy Across the Engine
  5. First Binary: Initializing an Empty Database File

Chapter 3: Storage Fundamentals — Disks, SSDs, and Pages

  1. How Disks and SSDs Actually Work
  2. Why Databases Think in Pages, Not Bytes
  3. Building Our Page Abstraction Layer
  4. Binary Data Layout: Endianness, Alignment, and Serialization
  5. Implementing the Storage Manager Interface

Chapter 4: File Format and Slotted Page Records

  1. Designing the Database File Format
  2. Slotted Page Layout: Headers, Directories, and Records
  3. Variable-Length Record Serialization
  4. Free-Space Management Within Pages
  5. Implementing Page Reads and Writes with Verification

Chapter 5: Buffer Pool Manager — Caching at Scale

  1. The Purpose of Buffer Pools and Why They Are Hard
  2. Pinning Semantics and Reference Counting
  3. Dirty Pages, Write Policies, and Flush Behavior
  4. Handling Memory Pressure and Eviction Under Load

Chapter 6: Logging, Durability, and Crash Recovery

  1. The Durability Problem and Why fsync Matters
  2. Write-Ahead Logging: Protocol and Invariants
  3. Designing Our Log Record Format
  4. Checkpointing Strategies and Implementation
  5. Crash Recovery: Redo and Undo in Practice

Chapter 7: Indexing — B-Trees, Hash Indexes, and Beyond

  1. Why Indexes Exist and What They Trade Away
  2. Implementing a B+ Tree from Scratch
  3. Page Splits, Merges, and Concurrency Considerations
  4. Hash Indexes for Equality Lookups
  5. LSM Trees and Alternative Index Architectures

Chapter 8: Transactions and Concurrency Control

  1. ACID Properties: What They Mean in Implementation Terms
  2. Locking Protocols and Two-Phase Locking
  3. Deadlock Detection and Prevention
  4. Latches vs Locks: Protecting Data Structures
  5. Implementing MVCC with Snapshot Isolation

Chapter 9: SQL Parsing and Semantic Analysis

  1. Designing Our SQL Subset and Grammar
  2. Lexical Analysis: Tokenizing SQL Input
  3. Recursive Descent Parsing and AST Construction
  4. Symbol Tables and Schema Resolution

Chapter 10: Query Planning and Cost-Based Optimization

  1. From AST to Logical Plan
  2. Physical Operators and Execution Strategies
  3. Building a Cost Model from Statistics
  4. Cardinality Estimation: Histograms and Selectivity
  5. Join Ordering and Query Rewrite Rules

Chapter 11: Query Execution Engine

  1. Operator Architecture: Iterators and Pipelining
  2. Table Scans and Index Scans
  3. Filtering, Projection, and Expression Evaluation
  4. Join Implementations: Nested-Loop, Hash, and Merge
  5. Aggregation and Grouping with Hash Tables

Chapter 12: Performance Engineering and Optimization

  1. Understanding Database Performance Characteristics
  2. I/O Patterns: Sequential Versus Random Access
  3. Cache Efficiency and Data Locality
  4. Vectorized Execution vs Tuple-at-a-Time Processing
  5. Profiling, Benchmarking, and Workload Generation
  6. Systematic Optimization: Finding and Fixing Bottlenecks

Chapter 13: Reliability, Correctness, and Failure Handling

  1. Crash Consistency: Guarantees and Invariants
  2. Torn Pages and Partial Writes
  3. Corruption Detection with Checksums and Validation
  4. Disk-Full Conditions and Safe Failure Modes
  5. Backup, Restore, and Integrity Verification

Chapter 14: Beyond Single Node — Distributed Database Concepts

  1. Why Single-Node Databases Eventually Hit Limits
  2. Replication Architectures: Leader-Follower and Multi-Leader
  3. Log Replication and Consistency Models
  4. Sharding and Distributed Query Execution
  5. Consensus, Fault Tolerance, and Rebalancing

Chapter 15: Conclusion — From Educational Engine to Production System

  1. Architecture Review: How All Subsystems Fit Together
  2. What We Simplified and Why Production Systems Differ
  3. Comparing Our Implementation to Real Databases
  4. Paths Forward: Extending Your Database Engine
  5. The Engineer’s Mindset for Building Reliable Systems

References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub