Chapter 6 · Part II — The learner

Training day

The network from Chapter 5 starts with random weights. This chapter is about the loop that makes them good — PPO, self-play against a shelf of frozen past selves — and about the run where win-rate climbed to 92%, cratered to 40%, and every dashboard said everything was fine.

The update problem

Picture the moment after data collection. You just played 48 games with the current policy. Some of them ended in wins, some in losses, and every decision along the way got recorded: the position, the menu, the chosen action, its probability at the time. Now you have to answer one question: how hard do you yank the probabilities?

The tempting answer is "hard" — this action preceded a win, crank it way up. The problem is that all 48 games were played by the old policy. The recorded probabilities, the visited positions, the whole dataset describes how the old policy behaves. If one update jumps the network far from where it was, the data that justified the jump no longer describes the network you now have. You've deployed a big-bang rewrite based on evidence gathered from the previous release, and your next batch of games gets played by a stranger.

Yank too gently and nothing improves. Yank too hard and you destroy what worked. Every deep RL algorithm is a stance on this dial.

PPO: nudge, but stay close

PPO — Proximal Policy Optimization — is the stance this project takes, in ppo_update in src/selfplay/trainer.py. Plainly: for each recorded decision, nudge the chosen action's probability up (if it preceded good outcomes) or down (if bad), but clip the size of any single update so the new policy stays near the policy that gathered the data. That's the "proximal" — nearby. Concretely, PPO looks at the ratio of new probability to old probability for each action, and if a nudge would push that ratio past 1 ± 0.2 (the clip=0.2 in the code), it stops paying gradient for going further. There is no reward for lurching.

Two more knobs live in the same function. The update runs several passes (4 epochs, minibatches of 256) over the batch, squeezing more learning from expensive-to-collect games. And an entropy bonus (coefficient 0.01) pays the policy a small reward for staying uncertain — entropy is a measure of how spread-out a probability distribution is, and keeping some spread means the policy keeps exploring instead of freezing onto one answer. Remember that word; it's the needle that moves in the war story.

One training day, concretely

The loop in src/selfplay/selfplay_train.py is three verbs, repeated ~120 times:

• Collect. Play 48 games in parallel through the VecPool (6 Node processes × 8 games each), the learner driving p1 against an opponent we'll meet in a second. Every learner decision is recorded as a transition. In self-play collection generally, both sides' decisions can be recorded — two trajectories per game for the price of one simulation.
• Update. One PPO update over the batch: 4 clipped epochs, plus the KL guard we'll get to.
• Eval. Every second iteration, a quick 60-game read against vs-random (±6 percentage points of noise at that sample size — a smoke test, not a verdict), and when it looks strong, a slow 200-game confirm.

By the start of the vectorized era each collect-update cycle took about 1.2 seconds — down from 14–18 seconds before vectorization. That ~12× speedup is what made everything in this chapter observable at all: the collapse only surfaced once long runs became cheap enough to actually run.

The opponent: a frozen league of past selves

Who does the learner play against? The obvious answer — itself, live — is a trap. If both players are the same network updating every iteration, the environment itself shifts under the learner every step. You're optimizing against a moving target that moves because you optimized. Learning becomes chasing your own tail: beat strategy A, opponent morphs, strategy A stops being tested, forget it, opponent rediscovers it, lose to it. Rock–paper–scissors forever.

So the opponent is a frozen league: a shelf of past snapshots of the learner. The run starts with one snapshot (the untrained net, gen_000) and every 6 iterations adds a frozen copy of the current learner. Each collection batch samples one league member at random to be p2. The opponent is stable within a batch (learnable signal), varied across batches (no overfitting to a single opponent), and historically deep (old strategies stay in the test set, so they can't be silently forgotten).

On top of that sits best-by-confirm checkpointing: the learner is only saved as the best checkpoint when a long 200-game confirm run beats the previous best's confirmed score. A hot streak on a noisy 60-game eval saves nothing. The last known-good build is never overwritten by a spot check.

The war story: the late win-rate collapse

Here is the chapter's heart, straight from docs/caveats.md. It happened twice — once as a mystery, once as a reproduction that revealed the cause.

The KL guard that said everything was fine

Here's the part worth a permanent place in your engineering brain. The PPO update has a safety metric: KL divergence, a standard measure of how far one probability distribution has moved from another — here, how much the policy's action probabilities shifted in one update. The code targets 0.02: if an update's measured KL exceeds that, the remaining epochs are skipped. A guard against exactly the kind of lurch that destroys policies.

During the collapse, the measured KL sat around 0.001 — twenty times under the alarm threshold — the entire time. The guard watched the policy destroy itself and reported calm at every step. How? The evals ran the policy greedy: always the argmax, the single highest-probability action. When two actions sit at 30.1% and 29.9%, a microscopic logit shift — nearly invisible to KL, which sums tiny probability changes across the whole distribution — swaps which one is the argmax. Behavior flips completely; probability distance barely moves. Many decisions, many near-ties, compounding over a 20-decision game: the played-out behavior drifted enormously while the guarded quantity stayed asleep.

Learning-rate decay, in one paragraph

The learning rate is the size of each weight nudge. The fixed run used 3e-4 forever; the fixed-collapse run uses 1e-4 with cosine decay down to 1e-5 across the 120 iterations — big steps early, when the weights are random and any direction is an improvement, shrinking smoothly to small steps late, when the policy is good and a big step can only break something delicate. The ML word is annealing, after the metallurgy: cool the metal slowly and the structure settles; quench it and it cracks. Early on you're roughing out the shape; by iteration 100 you're polishing, and you don't polish with a hammer.

What training can't fix

End-of-chapter honesty. This loop is magnificent at one thing: making the number you gave it go up. It was given "win the game" — a single ±1 reward per episode — and it will optimize exactly that, with the full creativity of millions of games, and nothing else. Two consequences are already looming.

First: one ±1 per ~20 decisions is brutally thin evidence. Which of those twenty clicks earned the win — the Fake Out on turn 1 or the lucky Rock Slide on turn 9? That's credit assignment, Chapter 7's problem, and the value head from Chapter 5 is about to earn its keep. Second, and darker: when the number itself stops meaning what you think it means — when win-rate against a saturated benchmark keeps looking fine while the policy quietly forgets how to set Trick Room — no amount of training discipline saves you. The collapse in this chapter announced itself with a cratering metric. Chapter 10 is about the failure that didn't.