Policy-Space Response Oracles in Practice

There is no book about policy-space response oracles. There is a survey, the original paper, and several dozen variant papers. None of them will tell you how many episodes per payoff cell you actually need, what happens when your best-response oracle undertrains, or whether the variant you are about to implement will beat plain self-play on your domain at the budget you can afford.

This book answers those questions by running the experiments. There is one PSRO loop — eighty-four lines, printed in full — and every algorithm in the book is that loop with different components plugged in. There is one companion repository, roughly five thousand lines, small enough that you can read every line. Every number in every table came out of a committed experiment, and every figure caption names the script, the seed, and the compute budget that produced it.

Which means the book reports results that are inconvenient. A diversity chapter that finds no benefit at 2.3× the wall-clock cost. A warm-start experiment that disproves the mechanism the chapter had proposed. A capstone domain whose first version turned out to be game-theoretically dead. And a flagship comparison of seven variants that refused to separate at all — until a second, larger budget point inverted it, turning the apparent winner into the worst method by a factor of four.

That last result is the book in miniature: a variant declared best at one budget can be worst at another, and nothing in a single-budget table tells you which you are looking at.

You will build Nash solvers from scratch, fictitious play, double oracle, a tabular best-response oracle, a from-scratch PPO oracle, five meta-strategy solvers, a parallel PSRO loop with bit-identical results at any worker count, a diversity-regularized oracle, a diagnostic dashboard, and a pursuit-evasion capstone solved end to end.

You should read it if you can train a PPO agent and are hazy on game theory. Parts I and II run on a laptop; Parts III and IV were run on one to eight V100s, and every experiment ships a smoke mode that finishes in under a minute.

Companion repository: github.com/aerodyneai/psro-in-practice