Leanpub Header

Skip to main content

Agentic QA

Building a Multi-Agent Testing Team with LLMs, Playwright and MCP

Agentic QA
This book is 100% completeLast updated on 2026-10-31

Go from “AI as a copilot” to a team of AI testing agents that analyze, design, run, triage and review tests, with a human test lead making the decisions. Hands-on, in TypeScript, and vendor-agnostic: Claude, GPT, Gemini or a local model, with tools shared through MCP.

Minimum price

$9.99

$19.99

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
About

About

About the Book

Agentic QA: Building a Multi-Agent Testing Team with LLMs, Playwright and MCP

Version 1.0: the complete book.

Build QA Crew, a team of AI agents that tests a real booking app: an Analyst, a Designer, an Executor, a Triager and a Reviewer. TypeScript, Playwright and MCP, vendor-neutral: Claude, GPT, Gemini or a local model through Ollama, by changing configuration, not code. Every chapter has working code and a tagged commit in the companion repository.

PART 1. FOUNDATIONS

1. From Copilot to Crew

2. The Anatomy of a QA Agent

3. Tools and MCP

4. Spec-Driven and Organic Driven Development

PART 2. BUILDING SINGLE QA AGENTS

5. The Analyst

6. The Designer

7. The Executor

8. The Triager

PART 3. MULTI-AGENT ORCHESTRATION

9. Orchestration Patterns for QA Teams

10. The Reviewer: Agents That Check Agents

11. QA Crew Inside Your IDE and Your Team

PART 4. PRODUCTION AND TRUST

12. Evaluating Your QA Agents

13. QA Crew in CI/CD

14. Testing Software That Has AI Inside

Final Words, Acknowledgments, three appendices (environment setup, mapping to agent frameworks, a prompt and rubric library) and references.

Real data, not promises: Claude, Gemini and a local model measured on the same tasks, and the mistakes they made, in the book.

Agents propose. Humans decide.

Author

About the Author

Enrique A. De Coss

Enrique A. De Coss is a Senior IT Leader in Quality Engineering with more than 20 years of experience building testing strategies and QA teams at companies such as Zurich Insurance, FICO, Kueski and Wizeline. He writes about the practical side of AI-assisted testing: what works, what breaks, and how to keep humans in charge.

Contents

Table of Contents

Preface

  1. Who this book is for
  2. What you will build
  3. How this book is organized
  4. How to use the code
  5. Part 1 — Foundations

From Copilot to Crew: Why Agentic QA

  1. Four ways to use AI in testing
  2. Where agents help today, and where they hurt
  3. You are the test lead now
  4. Meet QA Crew
  5. Setting up the companion project
  6. Summary

Anatomy of a QA Agent

  1. The five parts of an agent
  2. A vendor-neutral model layer
  3. The agent loop
  4. Structured output: results you can test
  5. Your first QA agent
  6. Testing the agent loop
  7. What you built
  8. Summary

Tools and MCP: Giving Agents Hands

  1. The problem with hand-written tools
  2. What MCP is
  3. Using an existing server: Playwright MCP
  4. Building qa-tools
  5. Connecting the agent to MCP
  6. One server, every assistant
  7. Tool safety
  8. Testing the tools
  9. What you built
  10. Summary

Where QA Plugs into Agentic Development: SDD and ODD

  1. Spec-Driven Development
  2. Organic Driven Development
  3. Choosing a workflow, and where QA fits
  4. One list of requirements
  5. The trace report
  6. What you built
  7. Summary
  8. Part 2 — Building Single QA Agents

The Analyst: From Requirements to Risks

  1. What the Analyst does, and what it does not
  2. Grounding: giving the Analyst a memory
  3. The risk map contract
  4. The Analyst’s instructions
  5. The model proposes, code checks, you decide
  6. Running the Analyst
  7. A real run
  8. Testing the Analyst
  9. What you built
  10. Summary

The Designer: Generating Test Cases You Would Actually Keep

  1. From risks to test cases
  2. Techniques as instructions
  3. Test data: ask code, not the model
  4. The test plan contract
  5. The Designer agent
  6. Curation: code keeps the plan honest
  7. Running the Designer
  8. A real run
  9. From the plan to your test case management tool
  10. Testing the Designer
  11. What you built
  12. Summary

The Executor: Agents That Write and Run Playwright Tests

  1. Conventions first: fixtures and page objects
  2. Quality gates: rules a test must pass before it exists
  3. A sandbox for generated tests
  4. The Executor’s procedure
  5. Trust, but verify
  6. Running the Executor
  7. A real run
  8. Exploratory testing with a charter
  9. Testing the Executor
  10. The crew so far
  11. What you built
  12. Summary

The Triager: From Red Build to Actionable Bug

  1. Five answers to one question
  2. Evidence first: what code collects
  3. The tools: the tracker, history and what we know
  4. The Triager’s rules
  5. Guards: when the answer and the evidence disagree
  6. A red build to triage
  7. Triaging the Executor’s findings
  8. Testing the Triager
  9. What you built
  10. Summary
  11. Part 3 — Multi-Agent Orchestration

Orchestration Patterns for QA Teams

  1. Five ways to organize a crew
  2. A run, on disk
  3. The crew’s four stages
  4. Gates: where the human decides
  5. A budget for the whole run
  6. The right model for each role
  7. Running the crew
  8. When things go wrong
  9. Fan-out: should test cases run in parallel?
  10. Testing the orchestrator
  11. What you built
  12. Summary

The Reviewer: Agents That Check Agents

  1. What a reviewer is for
  2. The rubric
  3. Code first: facts and suspicions
  4. The Reviewer
  5. The evaluator loop
  6. The crew catches TC-05
  7. Who reviews the reviewer?
  8. What a reviewer costs
  9. Testing the Reviewer
  10. What you built
  11. Summary

QA Crew Inside Your IDE and Your Team

  1. Three kinds of tools
  2. Asking the person, not the assistant
  3. Long work behind a quick tool
  4. Files as resources, routines as prompts
  5. Setting it up
  6. A session in the editor
  7. The bug, filed
  8. The crew in your team
  9. Testing the crew server
  10. What you built
  11. Summary
  12. Part 4 — Production and Trust

Evaluating Your QA Agents

  1. Tests, evals and why agents need both
  2. The golden set: written by a person, before the model runs
  3. Freeze the inputs
  4. Scorers: code first, again
  5. Trials and the scorecard
  6. Run it with the scripted agents
  7. The score that flatters
  8. The slow suite: the Executor
  9. Two models, one question: which one for which role?
  10. What an eval cannot tell you
  11. Testing the evals
  12. What you built
  13. Summary

QA Crew in CI/CD

  1. Three workflows, three kinds of cost
  2. Every push: the scripted crew
  3. Real models: the eval gate
  4. Runs nobody watches: retries, quotas, secrets
  5. A crew run with human gates in GitHub
  6. The human becomes the bottleneck
  7. Testing the pipeline
  8. What you built
  9. Summary

Testing Software That Has AI Inside

  1. The feature: a booking assistant
  2. What changes, and what does not
  3. Check the decision, not the words
  4. The question set
  5. Every trial, or it does not hold
  6. Run it with the scripted assistant
  7. Prompt injection: when data gives orders
  8. A judge for the prose
  9. A real model as the assistant
  10. Testing the tests
  11. What you built
  12. Summary

Final Words

  1. What the crew could not do
  2. Why the human stays
  3. Learn. This is not a trend.
  4. Thank you, and keep in touch

Acknowledgments

  1. A note on how this book was made

Appendix A — Setting Up Your Environment

  1. What you need
  2. Install
  3. Choose a model
  4. Keep costs low
  5. Connect the crew to your IDE
  6. Run it in CI
  7. Troubleshooting

Appendix B — Mapping QA Crew to Agent Frameworks

  1. The layers
  2. Concept by concept
  3. What to keep, whatever framework you use

Appendix C — Prompt and Rubric Library

  1. What the prompts have in common
  2. The Analyst (Chapter 5)
  3. The Designer (Chapter 6)
  4. The Executor (Chapter 7)
  5. The Triager (Chapter 8)
  6. The Reviewer (Chapter 10)
  7. The booking assistant (Chapter 14)
  8. The judge (Chapter 14)
  9. A template for your own QA agent

References

  1. Books on testing
  2. Chapter 3: Tools and MCP
  3. Chapter 4: Spec-Driven and Organic Driven Development
  4. Chapter 5: The Analyst
  5. Chapter 8: The Triager
  6. Chapter 9: Orchestration Patterns for QA Teams
  7. Chapter 10: The Reviewer
  8. Chapter 12: Evaluating Your QA Agents
  9. Chapter 14: Testing Software That Has AI Inside
  10. Appendix B: Mapping QA Crew to Agent Frameworks
  11. Tools used in the companion code

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub