Leanpub Header

Skip to main content
Go to Leanpub.com

Object Storage as Infrastructure

Building Databases, Queues, Git Systems, and Distributed Applications on S3-Compatible Storage

Object Storage as Infrastructure
This book is 100% completeLast updated on 2026-08-26
Steve Publications

What if S3 could be more than a place to put files? Build databases, queues, event logs, Git systems and distributed applications directly on object storage. Through production-grade Go implementations, this book explores what works, what breaks and how to design reliable systems around the real guarantees of S3-compatible storage.

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
197
Pages
About

About

About the Book

This book teaches experienced software engineers how to treat S3-compatible object storage as a first-class infrastructure primitive for building databases, message queues, event logs, version control systems, analytical platforms, and distributed applications. Through complete production-grade implementations in Go, you will learn to design around the strengths and constraints of object storage, reason about consistency and failure modes, and build reliable systems that scale from single servers to multi-region deployments. The goal is not to force every workload onto S3 but to understand the engineering boundaries of the abstraction so you can make deliberate architectural choices grounded in reality.

Share this book

Categories

StorageComputers and ProgrammingInfrastructure as CodeCloud ComputingGo

Feedback

Email the Author

Author

About the Author

Steve Publications

Steve is a technology professional with more than 20 years of experience in software development, server infrastructure, cybersecurity, vulnerability research and reverse engineering. Throughout his career, he has designed, secured, analyzed and tested complex software and infrastructure, with a particular focus on understanding how systems fail and how they can be made more secure.

Outside of work, Steve enjoys sharing knowledge with the technology community. He collaborates with researchers, industry experts and technology professionals to write practical books covering software development, cybersecurity, cloud computing, networking, DevOps, artificial intelligence and enterprise technologies. His books focus on practical learning through clear explanations, real-world examples and hands-on exercises. With more than two decades of industry experience, his goal is to help IT professionals, students and technology enthusiasts build useful skills and stay current in a rapidly changing industry.

We believe readers deserve to know how our books are created. Most of our authors are not native English speakers, so we use AI to help translate, proofread manuscripts, fix grammar, improve sentence structure and make technical explanations easier to read. AI is used as an editing tool only. It does not replace the research, technical knowledge or hands-on experience behind our books. Some of our authors also prefer to remain anonymous for privacy or professional reasons. In those cases, we publish their work under a different name. The author's name may be different, but the quality of the content and our review process remain the same.

Every book is written, reviewed and maintained by experienced technology professionals, with contributions from our private technical community of more than 420 engineers and researchers. We spend far more time validating technical accuracy and keeping our content up to date than generating text. We are always interested in working with experienced professionals who have deep expertise in a particular technology or domain. If you would like to publish a book with us or help review an existing manuscript, we'd love to hear from you. Send us a message describing your area of expertise. We are especially interested in niche technologies, specialized skills and emerging topics that are underrepresented in existing technical literature.

If you look through the contents of our books, you'll see practical examples, detailed explanations and material that is regularly updated. Our goal is to publish books that professionals can actually rely on, not low-effort AI-generated content. If you ever feel that one of our books does not meet that standard, Leanpub offers a 60-day money-back guarantee. Feel free to request a refund if you are not satisfied with your purchase.

Contents

Table of Contents

Building Databases, Queues, Git Systems, and Distributed Applications on S3-Compatible Storage

Introduction

Chapter 1: The Object Storage Primitive

  1. What Object Storage Is and Is Not
  2. The S3 API Surface: Objects, Keys, Prefixes, Metadata
  3. Core Operations: PUT, GET, HEAD, DELETE, LIST, Multipart Upload
  4. Consistency Guarantees and Their Evolution
  5. Durability, Availability, and Replication Strategies
  6. Authentication, Authorization, and Security Primitives
  7. Object Storage Versus Block File Database and Queue Abstractions

Chapter 2: Designing Around Constraints

  1. Latency Throughput and Cost Realities
  2. Immutable Objects and Update Semantics
  3. Expensive Listing Operations and Namespace Design
  4. Conditional Writes and Optimistic Concurrency
  5. Metadata Layers and Indexing Strategies
  6. Handling Partial Failures and Network Partitions
  7. The Compute Separation Principle

Chapter 3: S3-Compatible Implementations and Ecosystem

  1. AWS S3 Architecture and Behavior
  2. Open Source Implementations: MinIO Ceph RGW
  3. Cloud Provider Variants: GCS Azure Blob DigitalOcean Spaces
  4. On-Premise and Air-Gapped Deployments
  5. API Compatibility Gaps and Gotchas
  6. Choosing and Evaluating an Implementation

Chapter 4: Content-Addressable Storage and Immutable Data Structures

  1. Hash-Based Object Identification
  2. Deduplication and Size Savings
  3. Object Layout and Key Design
  4. Implementation: A Content-Addressable Blob Store in Go
  5. Testing and Verification
  6. Performance Characteristics and Tuning

Chapter 5: Append-Only Event Logs on Object Storage

  1. Log Semantics and Ordering Guarantees
  2. Segment Design and Batching
  3. Sequence Numbers and Fencing
  4. Implementation: A Partitioned Event Log in Go
  5. Compaction and Retention
  6. Recovery from Partial Writes and Corruption

Chapter 6: Durable Queues and Message Streams on Object Storage

  1. Queue Semantics and Delivery Guarantees
  2. Visibility Tokens and Acknowledgement Patterns
  3. Implementation: A Durable Queue on Object Storage
  4. Partitioned Streams and Consumer Groups
  5. Exactly-Once Considerations and Idempotency
  6. Performance Comparison with Kafka and Traditional Brokers

Chapter 7: Key-Value Stores and Metadata Databases on Object Storage

  1. Key-Value Semantics on Immutable Objects
  2. SSTable-Inspired Layout and LSM Concepts
  3. Indexes and Bloom Filters for Efficient Lookup
  4. Implementation: A Lightweight KV Store in Go
  5. Read Path Optimization and Caching Strategies
  6. Compaction Snapshots and Garbage Collection

Chapter 8: Analytical Data Stores and Query Execution on Object Storage

  1. Columnar Formats and Parquet on Object Storage
  2. Partitioning Strategies for Query Performance
  3. Predicate Pushdown and Filter Semantics
  4. Implementation: A Simple Analytical Query Engine in Go
  5. Schema Evolution and Compatibility
  6. Metadata Catalogs and Table Management

Chapter 9: Git-Compatible Storage Backends on Object Storage

  1. Git Objects References and the DAG
  2. Immutable Storage and Deduplication in Git
  3. Packfiles and Delta Compression
  4. Designing an S3-Compatible Git Backend
  5. Ref Consistency and Concurrent Updates
  6. Implementation: A Git Service with Object Storage Backend

Chapter 10: Distributed Coordination on Object Storage

  1. Leases and Leader Election Patterns
  2. Distributed Locking with Conditional Writes
  3. Fencing Tokens and Session Management
  4. Configuration Stores and Feature Flags
  5. When to Use Object Storage Versus Dedicated Coordination Services
  6. Implementation: A Lease-Based Coordinator in Go

Chapter 11: Performance Engineering at Scale

  1. Measuring What Matters: RPS Throughput Tail Latency Cost
  2. Request Batching and Parallel Upload Strategies
  3. Connection Management and Asynchronous I/O
  4. Caching Architectures: Memory Local Disk CDN Layers
  5. Compression Serialization and Object Sizing
  6. Benchmarking Methodology and Realistic Tests

Chapter 12: Reliability Failure Handling and Recovery

  1. Taxonomy of Failures in Object Storage Systems
  2. Idempotent Operations and Safe Retries
  3. Handling Corrupted Objects and Integrity Verification
  4. Recovery Procedures for Each Component Type
  5. Fault Injection and Property-Based Testing
  6. Disaster Recovery and Multi-Region Architectures

Chapter 13: Security Access Control and Compliance

  1. Identity and Access Management Patterns
  2. Temporary Credentials and Workload Identity
  3. Encryption in Transit and at Rest
  4. Object Locking and Immutability Controls
  5. Tenant Isolation in Multi-Tenant Systems
  6. Security Auditing and Common Misconfigurations

Chapter 14: Observability Operations and Production Deployment

  1. Structured Logging and Request Tracing
  2. Metrics SLIs and SLOs for Object Storage Systems
  3. Capacity Planning and Lifecycle Management
  4. Backup Restore and Migration Procedures
  5. Incident Response and Operational Runbooks

Chapter 15: Case Studies from the Field

  1. Cloud-Native Databases Built on Object Storage
  2. Open-Source Projects Using S3 as Primary Storage
  3. Streaming and Queue Systems with Object Backends
  4. Git Hosting Platforms and Large-Scale Repositories
  5. Lessons Learned and Common Pitfalls
  6. What These Architectures Teach Us

Chapter 16: A Production Reference Architecture

  1. Architecture Overview and Component Interactions
  2. Storage Layout Across Services
  3. Implementation: A Multi-Service System in Go
  4. Deployment Configuration and Infrastructure Setup
  5. Operational Procedures and Monitoring
  6. Scaling Upgrading and Evolving the System

Conclusion

References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub