Engineering AI-Powered Browser Extensions and Autonomous Browser Agents
Chapter 1: The Browser as an Agent: A New Computing Paradigm
- The Passive Web
- What It Means for Software to Think
- Why the Browser Is the Agent’s Natural Habitat
- What This Book Builds
- How to Read This Book
Chapter 2: A Brief History of Talking to Browsers
- Scripting the Browser: The Macro Era
- The Test Automation Era
- The Developer Era: CDP, Puppeteer and Playwright
- The Agent Era
- Scripted vs. Agentic: The Distinction That Matters
Chapter 3: Inside the Browser: Chromium as an Operating System
- The Multi-Process Architecture
- The Rendering Pipeline
- The JavaScript Engine
- The DOM, the CSSOM and the Accessibility Tree
- The Security Model
Chapter 4: The WebExtensions Platform
- From NPAPI to WebExtensions
- Manifest V3 in Depth
- Content Scripts and Their Worlds
- Background Service Workers
- Messaging Architecture
- Storage, Identity and Boundaries
Chapter 5: Controlling the Browser from Outside: CDP and the Automation Stack
- The Chrome DevTools Protocol
- WebDriver and the W3C Standard
- Playwright Internals
- Puppeteer and Its Lineage
- Choosing a Control Plane
Chapter 6: Language Models for Browser Engineers
- The Transformer, Practically
- Inference: Cloud, Local and Hybrid
- The Economics of Tokens
- Model Selection for Agents
- Multimodal Models
Chapter 7: Context Engineering: Turning Web Pages into Something a Model Can Use
- The Page as Data
- Serialization Strategies
- Token Budgeting and Pruning
- Vision as Context
- Structured Page Schemas
Chapter 8: Tool Use: Giving Models Hands
- How Function Calling Actually Works
- Designing Tool Schemas for the Web
- Structured Outputs and JSON Constraints
- Multi-Step Tool Orchestration
- Error Recovery in Tool Use
Chapter 9: The Agent Loop: From Prompt to Autonomous Behavior
- ReAct and the Observe-Think-Act Pattern
- Planning and Task Decomposition
- State Management and the Blackboard
- Reflection and Self-Correction
- Termination, Policies and Guardrails
Chapter 10: DOM-Driven Agents
- The Case for the DOM
- DOM Perception in Practice
- Element Targeting and the Identifier Problem
- Dynamic Content, SPAs and Iframes
- The DOM Agent Implementation Pattern
- Strengths and Failure Modes
Chapter 11: Vision-Based and Hybrid Perception
- The Screenshot Pipeline
- Visual Grounding and Coordinate Mapping
- Multimodal LLMs as Perceivers
- Hybrid Perception Architectures
- When Each Modality Wins
Chapter 12: Memory and Long-Horizon Agency
- The Memory Taxonomies That Matter
- Retrieval-Augmented Generation for Agents
- Checkpointing and Resumption
- Context Compression and Summarization
- Building a Memory Backend
Chapter 13: Multi-Agent and Hierarchical Systems
- Why Decompose
- Supervisor-Worker and Hierarchical Patterns
- Multi-Agent Communication
- Orchestrating Shared Browser Sessions
- The Case Against Multi-Agent
Chapter 14: Security, Trust and Responsible Autonomy
- Prompt Injection Through the Web
- The Hostile Page Problem
- Credential and Session Security
- Human-in-the-Loop by Design
- Privacy, Compliance and Responsible Autonomy
Chapter 15: Project Alpha: The Minimal AI Browser Extension
- Toolchain and Project Structure
- The Extension Skeleton
- Page Context Extraction
- LLM Integration and Streaming
- The UI and User Experience
- Testing, Packaging and Loading
Chapter 16: Project Beta: The Context-Aware Copilot
- Architecture Overview
- The Deep Page Understanding Pipeline
- Multi-Turn Conversation with Page Context
- Session Memory and Persistence
- Page Change Detection and Live Context
- Streaming, Cancellation and Performance
Chapter 17: Project Gamma: The Interaction Agent
- Action Space Design
- The Perception Snapshot
- The Agent Loop Implementation
- Form Filling as a Subsystem
- Verification and Recovery
- Safety Gates
Chapter 18: Project Delta: The Research Agent
- Why research is hard
- Architecture overview
- The five phases
- Phase 1: Planning
- Phase 2: Discovery
- Phase 3: Reading
- The content script
- Phase 4: Synthesis
- Phase 5: Reporting
- State management and checkpointing
- The orchestrator
- The side panel UI
- Token budgeting
- Error handling and recovery
- The complete file list
- Build and run
- Testing Delta
- Limitations
- Where Delta fits
Chapter 19: Project Epsilon: The Production Platform
- What changes when you go to production
- System architecture
- The LLM gateway
- The headless browser reader
- The backend API
- Authentication
- Rate limiting and cost control
- The database
- The orchestrator (server-side)
- The thin client extension
- Observability
- Security in production
- Deployment
- The complete file list
- Build and run
- Testing Epsilon
- Limitations
- Where Epsilon fits
Conclusion: The Next Decade of Browsers
- The architectural lessons
- The projects, in one paragraph each
- The next frontier
- A note on responsibility
- The code is yours