Preface
- Who this book is for
- What you will build
- How this book is organized
- How to use the code
- Part 1 — Foundations
From Copilot to Crew: Why Agentic QA
- Four ways to use AI in testing
- Where agents help today, and where they hurt
- You are the test lead now
- Meet QA Crew
- Setting up the companion project
- Summary
Anatomy of a QA Agent
- The five parts of an agent
- A vendor-neutral model layer
- The agent loop
- Structured output: results you can test
- Your first QA agent
- Testing the agent loop
- What you built
- Summary
Tools and MCP: Giving Agents Hands
- The problem with hand-written tools
- What MCP is
- Using an existing server: Playwright MCP
- Building qa-tools
- Connecting the agent to MCP
- One server, every assistant
- Tool safety
- Testing the tools
- What you built
- Summary
Where QA Plugs into Agentic Development: SDD and ODD
- Spec-Driven Development
- Organic Driven Development
- Choosing a workflow, and where QA fits
- One list of requirements
- The trace report
- What you built
- Summary
- Part 2 — Building Single QA Agents
The Analyst: From Requirements to Risks
- What the Analyst does, and what it does not
- Grounding: giving the Analyst a memory
- The risk map contract
- The Analyst’s instructions
- The model proposes, code checks, you decide
- Running the Analyst
- A real run
- Testing the Analyst
- What you built
- Summary
The Designer: Generating Test Cases You Would Actually Keep
- From risks to test cases
- Techniques as instructions
- Test data: ask code, not the model
- The test plan contract
- The Designer agent
- Curation: code keeps the plan honest
- Running the Designer
- A real run
- From the plan to your test case management tool
- Testing the Designer
- What you built
- Summary
The Executor: Agents That Write and Run Playwright Tests
- Conventions first: fixtures and page objects
- Quality gates: rules a test must pass before it exists
- A sandbox for generated tests
- The Executor’s procedure
- Trust, but verify
- Running the Executor
- A real run
- Exploratory testing with a charter
- Testing the Executor
- The crew so far
- What you built
- Summary
The Triager: From Red Build to Actionable Bug
- Five answers to one question
- Evidence first: what code collects
- The tools: the tracker, history and what we know
- The Triager’s rules
- Guards: when the answer and the evidence disagree
- A red build to triage
- Triaging the Executor’s findings
- Testing the Triager
- What you built
- Summary
- Part 3 — Multi-Agent Orchestration
Orchestration Patterns for QA Teams
- Five ways to organize a crew
- A run, on disk
- The crew’s four stages
- Gates: where the human decides
- A budget for the whole run
- The right model for each role
- Running the crew
- When things go wrong
- Fan-out: should test cases run in parallel?
- Testing the orchestrator
- What you built
- Summary
The Reviewer: Agents That Check Agents
- What a reviewer is for
- The rubric
- Code first: facts and suspicions
- The Reviewer
- The evaluator loop
- The crew catches TC-05
- Who reviews the reviewer?
- What a reviewer costs
- Testing the Reviewer
- What you built
- Summary
QA Crew Inside Your IDE and Your Team
- Three kinds of tools
- Asking the person, not the assistant
- Long work behind a quick tool
- Files as resources, routines as prompts
- Setting it up
- A session in the editor
- The bug, filed
- The crew in your team
- Testing the crew server
- What you built
- Summary
- Part 4 — Production and Trust
Evaluating Your QA Agents
- Tests, evals and why agents need both
- The golden set: written by a person, before the model runs
- Freeze the inputs
- Scorers: code first, again
- Trials and the scorecard
- Run it with the scripted agents
- The score that flatters
- The slow suite: the Executor
- Two models, one question: which one for which role?
- What an eval cannot tell you
- Testing the evals
- What you built
- Summary
QA Crew in CI/CD
- Three workflows, three kinds of cost
- Every push: the scripted crew
- Real models: the eval gate
- Runs nobody watches: retries, quotas, secrets
- A crew run with human gates in GitHub
- The human becomes the bottleneck
- Testing the pipeline
- What you built
- Summary
Testing Software That Has AI Inside
- The feature: a booking assistant
- What changes, and what does not
- Check the decision, not the words
- The question set
- Every trial, or it does not hold
- Run it with the scripted assistant
- Prompt injection: when data gives orders
- A judge for the prose
- A real model as the assistant
- Testing the tests
- What you built
- Summary
Final Words
- What the crew could not do
- Why the human stays
- Learn. This is not a trend.
- Thank you, and keep in touch
Acknowledgments
- A note on how this book was made
Appendix A — Setting Up Your Environment
- What you need
- Install
- Choose a model
- Keep costs low
- Connect the crew to your IDE
- Run it in CI
- Troubleshooting
Appendix B — Mapping QA Crew to Agent Frameworks
- The layers
- Concept by concept
- What to keep, whatever framework you use
Appendix C — Prompt and Rubric Library
- What the prompts have in common
- The Analyst (Chapter 5)
- The Designer (Chapter 6)
- The Executor (Chapter 7)
- The Triager (Chapter 8)
- The Reviewer (Chapter 10)
- The booking assistant (Chapter 14)
- The judge (Chapter 14)
- A template for your own QA agent
References
- Books on testing
- Chapter 3: Tools and MCP
- Chapter 4: Spec-Driven and Organic Driven Development
- Chapter 5: The Analyst
- Chapter 8: The Triager
- Chapter 9: Orchestration Patterns for QA Teams
- Chapter 10: The Reviewer
- Chapter 12: Evaluating Your QA Agents
- Chapter 14: Testing Software That Has AI Inside
- Appendix B: Mapping QA Crew to Agent Frameworks
- Tools used in the companion code