Solving The N+1 Problem In PHP
Solving The N+1 Problem In PHP
About the Book
The web page was taking three hours to load.
Let that sink in for a moment: three hours.
The code backing the page was a reporting script written in PHP, attempting to build 40,000 entity objects from 2 million database rows. Indeed, sometimes the page never loaded at all. At least the client could tell when it failed, because a big ugly error popped up in the browser window. Unfortunately that was also around the three hour point. The client was understandably frustrated.
It turned out the cause was not that the database was unoptimized. It was not the number of rows being returned. It was not the number of objects being built.No, the cause was the high volume of queries being issued from the reporting script.
When I say "high volume", I mean two hundred thousand and one queries. These 200,000+ queries were used to build each root entity along with each of its five associated collections and entities, all as a result of how the script was assembling the entity objects. No wonder the script was was taking three hours to finish!
After discovering this, I was able to apply the techniques presented in this book to refactor how the entity objects were assembled. Over the course of a few days of refactoring I was able to reduce the load time from 3+ hours to just under 6 minutes. That's a 30x improvement, and while still slow by typical standards, it was much more bearable for the client. (Later refactorings improved the load time even further.)
The causes leading to the terrible performance of that reporting script have a name: The N+1 Problem. The problem goes by various aliases, such as the "1+N problem", "N+1 queries", "SELECT N+1", and so on, but they all refer to the same thing.
While I mention the N+1 problem more than once in Modernizing Legacy Applications in PHP, I did not address a solution directly. This book remedies that. Herein, we will discuss the specifics of the N+1 problem:
- what the N+1 problem is,
- how to diagnose and discover the problem in your application,
- why the problem occurs, and
- how to solve the problem in your codebase using plain old PHP (no ORMs needed!).
If your application code has database performance issues, or if you want to learn how to prevent creating an N+1 problem in your codebase, buy this book today!
About the Author
Table of Contents
-
1. Introduction
- The Slow Reporting Script
- Example Application
-
2. Defining The N+1 Problem
- The Problem By Example
- The Problem Defined
- Performance Implications
-
3. Diagnosing N+1 Problems
- Codebase Examination
- Log/Profile Examination
- Easier Query Examination
-
4. Why Does The N+1 Problem Occur?
- CRUD Is The Problem
- BREAD Is The Answer
-
5. Solving N+1 With A Single Query
- Handling “Many-To-One” Relationships
- Handling “One-To-One” Relationships
- Combining “Many-To-One” and “One-To-One”
- Handling “To-Many” Relationships
- Single-Query Fails With “To-Many” Relationships
-
6. Solving N+1 With Query-And-Stitch
- Tradeoffs In Query-And-Stitch
- Handling “One-To-Many” Relationships
- Handling “Many-To-Many” Relationships
- Handling “Many-To-One” Relationships
- Handling “One-To-One” Relationships
- Performance Implications
-
7. Refactoring The Codebase To Remove N+1 Problems
- Prerequisites
- The Process
- N+1 In A Single Location
- N+1 Across Separated Locations
- Test, Commit, Push, Notify QA
- Do … While
-
8. Common Questions
- Can I Combine Single-Query With Query-And-Stitch?
-
Can I Use Sub-Selects Instead Of The
IN()Operator? - Should I Use A Transaction With Query-And-Stitch?
- Can I Just Use An ORM?
-
Appendix
- SQL Schema
- PHP Classes
- Colophon
- About the Author
Authors have earned$9,690,775writing, publishing and selling on Leanpub, earning 80% royalties while saving up to 25 million pounds of CO2 and up to 46,000 trees.
Learn more about writing on Leanpub
The Leanpub 45-day 100% Happiness Guarantee
Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers), EPUB (for phones and tablets) and MOBI (for Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
Top Books
- #1
Ansible for Kubernetes
Jeff GeerlingAnsible is a powerful infrastructure automation tool. Kubernetes is a powerful application deployment platform. Learn how to use these tools to automate massively-scalable, highly-available infrastructure.
- #2
Practical FP in Scala: A hands-on approach
Gabriel VolpeA practical book aimed for those familiar with functional programming in Scala who are yet not confident about architecting an application from scratch.
Together, we will develop a purely functional application using the best libraries in the Cats ecosystem, while learning about design patterns and best practices.
- #3
Functional Design and Architecture
Alexander GraninSoftware Design in Functional Programming, Design Patterns and Practices, Methodologies and Application Architectures. How to build real software in Haskell with less efforts and low risks. The first complete source of knowledge.
- #4
Production Haskell
Matt ParsonsAre you excited about Haskell, but don't know where to begin? Are you thrilled by the technical advantages, but worried about the unknown pitfalls? This book has you covered.
- #5
Tame your Work Flow
Steve Tendon and Daniel DoironDo you need a high performance enterprise governance approach improving management, execution and delivery while dealing with multiple projects/products, events, stakeholders and teams? Giving you better bottom line results, faster time to market, less work, better predictability, happier employees, and delighted clients? Then learn about TameFlow!
- #6
Ansible for DevOps
Jeff GeerlingAnsible is a simple, but powerful, server and configuration management tool. Learn to use Ansible effectively, whether you manage one server—or thousands.
- #7
Machine Learning Engineering
Andriy Burkov"If you intend to use machine learning to solve business problems at scale, I'm delighted you got your hands on this book."
—Cassie Kozyrkov, Chief Decision Scientist at Google
"Foundational work about the reality of building machine learning models in production."
—Karolis Urbonas, Head of Machine Learning and Science at Amazon
- #8
C++ Best Practices
Jason TurnerLevel up your C++, get the tools working for you, eliminate common problems, and move on to more exciting things!
- #9
Composing Software
Eric ElliottAll software design is composition: the act of breaking complex problems down into smaller problems and composing those solutions. Most developers have a limited understanding of compositional techniques. It's time for that to change.
- #10
El Manual del Manager
Keyvan Akbary, Félix López, and Álvaro Salazar¿Has deseado alguna vez el haber tenido una buena introducción al rol del Engineering Manager? En este libro aprenderás lo necesario para ejercer el rol de una manera efectiva: Expectativas y Responsabilidades del Rol, 1-1s, Ayudar a Crecer, Objetivos, Planes de Carrera, Cultura, Feedback, Contratación, Cultura de Producto y mucho más.
Top Bundles
- #1
The Tester's Library
8 Books
The Tester's Library consists of eight five-star books that every software tester should read and re-read. As bound books, this collection would cost over $200. Even as e-books, their price would exceed $80, but in this bundle, their cost is only $49.99. Here are the books, and why they should be in your library: Perfect Software and Other... - #2
agile
11 Books
In this bundle, you will find 10 different agile books. They are about different aspects of being agile. - finding a job - doing coding dojo's - Retrospectives - Personal kanban - a non-typical coaching book and even a book that gives you an insight in the lives of some agile people. - #3
WTFlop 6M + HU - Beta Bundle
6 Books
- #4
Marionette.js A to Z
3 Books
Marionette.js has become a popular Backbone-based javascript library, because it helps you avoid boiler-plate code and focus on what makes your app special. But how do you get started?Whether you want to finally get started with dynamic client-side programming (like one reader did), or are an experienced front-end dev who has inherited a... - #5
Build A Better Backbone App
3 Books
The best way to learn new development skills is through experience, but that takes time you don't have.Get the best of both worlds with this bundle: you'll learn how to produce modern web applications by learning from experienced developers like Derick Bailey and David Sulc. BackboneJS is one of the favorite tools on the web today, but it... - #6
General Systems Thinker Bundle
5 Books
The General Systems Thinker Bundle is just that: a bundle of five books to advance the reader one giant step toward improved thinking, based on General Systems principles. Four of the books are the complete General Systems Series. The fifth is fictional piece which shows some general systems thinkers in action. It's a mystery in which a group of... - #7
Experiential Learning Bundle
4 Books
This bundle provides all four volumes of the popular Experiential Learning Series at a savings of $20 over the price if purchased separately. - #8
Ansible Bundle
2 Books
After getting up and running with Ansible in Jeff Geerling's Ansible for DevOps, strengthen your skills managing tens to thousands of instances and services in Amazon's AWS cloud with Yan Kurniawan's Ansible for AWS. - #9
Learn ECMAScript 6 inside and out
2 Books
For any technology, it helps to get multiple points of view on the functionality to get the best possible understanding. For ECMAScript 6/2015, no two resources are recommended more frequently thanExploring ES6by Dr. Axel Rauschmayer andUnderstanding ECMAScript 6by Nicholas C. Zakas. These two points of view, investigating the specification and... - #10
Software architecture, for systems old and new
2 Books
This bundle includes books about hands-on software architecture.
