Leanpub Header

Skip to main content
Go to Leanpub.com

Kafka Taxi

Learn Kafka by building a ride-sharing system, breaking it, and measuring every fix

Kafka Taxi
Lakshmi Narasimhan

Build a ride-sharing backend, put realistic load on it, and measure every fix, cheapest first. Find out where Kafka earns its place, and where it does not.

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
About

About

About the Book

Most Kafka books start with brokers, topics and partitions. This one starts with a problem.

We build the backend of a ride-sharing system, like Uber, and call it Kafka Taxi. Riders ask for rides, drivers send their location every few seconds, and the system matches them. The first version is one Python service and one Postgres database, and it is a good design: correct, tested and fast.

Then we put realistic load on it and watch it fail. We measure why. We try the cheaper fixes first: a bigger server, batched writes, Postgres notifications, Redis Streams and read replicas. Kafka only appears when the measurements show what it gives us that the others do not. Every number in the book comes from code you can run on your laptop.

This book also tells you when not to use Kafka. The most useful thing you can take from it is the ability to look at your own system and decide which side of that line it is on.

It is written in plain English for working backend engineers. You need to know HTTP and SQL. You do not need any background in distributed systems.

All seven chapters and the appendix are written. The appendix lists the exact versions and settings behind every measurement. When those versions change, I will update the book, and you get every update for free.

Companion code: https://github.com/badri/kafka-taxi-code

Share this book

Feedback

Contact the author

Author

About the Author

Lakshmi Narasimhan

I’m Lakshmi Narasimhan. Principal SRE by day, builder-writer-polymath by everything else.

I help developers build, deploy, and distribute their SaaS without hiring a team. I also run Stacksweller (Substack scheduling + analytics) and Supabyoi (managed self-hosted Supabase), write fiction under a pen name, play Carnatic violin, and occasionally publish research on Carnatic ML when the day pretends to end.

Contents

Table of Contents

Introduction

  1. What we are going to build
  2. The plan
  3. Why build something and then break it?
  4. This book will also tell you when not to use Kafka
  5. Who this book is for
  6. What you need

The Monolith

  1. Getting it running
  2. What the business needs
  3. Four tables
  4. A driver’s day in two endpoints
  5. Dispatch
  6. Does it work?
  7. Two riders, one driver
  8. Is it fast?
  9. What I left out on purpose
  10. One number we have not tested

The Bottleneck

  1. Realistic load
  2. A note about the hardware
  3. Five terms before we start
  4. The ladder
  5. What is the bottleneck?
  6. Why every ping costs so much
  7. The settings a good engineer tries first
  8. Why dropping the index does not work
  9. “Just buy a bigger server”
  10. “Then use a different database”
  11. The numbers at full scale
  12. The problem is the shape, not taxis
  13. So what do we need?

Changing the Shape of the Write

  1. First option: batched appends in Postgres
  2. Who else wants the positions?
  3. Fan-out attempt 1: LISTEN / NOTIFY
  4. Fan-out attempt 2: let consumers poll the table
  5. Fan-out attempt 3: Redis Streams
  6. What about read replicas?
  7. Someone had this problem before, at a much larger scale
  8. Kafka
  9. What Kafka costs

Producers and Topics

  1. Two kinds of fact
  2. What goes in a position event
  3. The producer is chapter 4’s buffer, already written
  4. Where did the fsync go?
  5. The key decides everything else
  6. Ride events can’t be sent from the request handler
  7. Where that leaves us

Partitioning and Consumer Groups

  1. What moves, and what stays in Postgres
  2. Consumer groups
  3. When members come and go
  4. Dispatch does not join a group
  5. Building the locality ourselves
  6. The claim stays in Postgres
  7. Chapter 3’s ladder, again
  8. The whole city
  9. What I left out, and what comes next

Ordering and Exactly-Once

  1. Three delivery promises
  2. The billing consumer
  3. Commit after the charge
  4. Make the charge safe to repeat
  5. Order is the consumer’s job too
  6. What about Kafka’s exactly-once?
  7. What three brokers buy
  8. The decision, as a checklist

Appendix: Versions and Settings

  1. Versions
  2. Postgres
  3. The producer
  4. Topics
  5. Consumers and the relay
  6. Broker defaults that mattered
  7. Switches in the code

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub