Chapter 7 · Part II — The learner

The credit problem

A game is about twenty decisions, and the only feedback is one number at the very end: +1 or −1. This chapter is about how the trainer figures out which of those twenty decisions deserved it — and about the time a carefully designed penalty turned out to do nothing at all.

Twenty decisions, one number

Recall the shape of the training signal from Chapter 3. The reward for an entire game — one episode, roughly twenty decisions after the bridge auto-plays the forced ones — is a single ±1 delivered at the final turn. Every turn before that, the reward is exactly zero. Not "small." Zero.

Now put yourself in the trainer's shoes. A game just ended +1. Somewhere in those twenty decisions was the turn-2 Trick Room that flipped the speed order and won the whole endgame. Also somewhere in there was a pointless turn-5 switch that accomplished nothing. Both decisions sit in a trajectory that ended +1, so both get pulled toward "do this more." The signal is real — over millions of games, good moves appear in winning games slightly more often than bad ones do — but it is diluted twenty ways and drowned in luck. This is the credit assignment problem, and it is the central difficulty of reinforcement learning with sparse rewards.

Discounting, briefly

One knob you should know exists: the discount factor γ. Rewards are shrunk slightly for every step of distance between a decision and the payoff — a move ten turns before the win gets credit multiplied by γ¹⁰. In this project γ = 0.997 (trainer.py), deliberately close to 1: games are short, and the earliest decision of all — the pick-4 draft at team preview — matters as much as anything that happens later, so its credit must arrive nearly undiscounted. At γ = 0.997, a reward twenty turns away still keeps about 94% of its value. Discounting here is a gentle tiebreaker toward faster wins, not a real horizon.

Fix #1 — PBRS: pay for progress, not just for winning

The first fix is potential-based reward shaping (PBRS), implemented in the potential() and assign_returns() functions of src/selfplay/trainer.py. The idea: invent a position score Φ — a single number for "how good is this position for me, right now." This project's Φ is a hand-written sum of four-and-a-bit terms:

With Φ in hand, every turn's reward gets a shaping term added: Φ(state after) − Φ(state before), scaled by a weight (SHAPE = 0.5). Do something that improves the position and you get paid that turn. Set Trick Room while slow and the speed bits flip: an immediate credit of about +0.6, landing on the exact decision that earned it, instead of a diluted share of a ±1 fifteen turns later.

Here is the subtle point — and it is the entire reason the word "potential-based" matters. Credit is only ever given as a difference of the position score. Everything you gain on entering a good state, you give back the moment you leave it: enter a Trick-Room-up position, collect +0.6; when Trick Room expires four turns later, Φ drops and you pay it back. Sum the shaping terms over any full game and they telescope — the middle terms cancel pairwise, leaving only Φ(end) − Φ(start), which is a constant that doesn't depend on how you played. So no strategy can farm the shaping. The best strategy under the shaped reward is provably the same as the best strategy under the real ±1 (this is a theorem, due to Ng, Harada and Russell). Φ is a denser progress bar toward the same goal, not a different goal.

Contrast that with the naive version: "+0.1 every time you KO something." That is not a difference of anything, so it doesn't cancel — a policy can learn to farm KOs against sacrificial fodder, drag games out to collect bonuses, and drift away from actually winning. Potential differences structurally can't be farmed. That's the trade: you get to inject a coaching opinion into training, and the math guarantees the opinion can only change how fast the policy learns, not what it ultimately converges to.

Measured effect: the shaped-vs-unshaped ablation rated 83 vs 64 Elo — about +19 Elo for shaping, per docs/roadmap.md. Useful, real, not enormous. The bigger prize was still to come.

Fix #2 — GAE: blending reality with the critic

The second fix attacks a different enemy: noise. Recall from Chapter 5 that the network has a value head — a side output that estimates, from any position, how likely we are to win from here. In RL jargon that estimator is called the critic. The learning update in PPO doesn't actually push on raw returns; it pushes on the advantage of each move: how much better did things go after this decision than the value head expected? Chose a move from a 50/50 position and ended up clearly winning — positive advantage, do it more. Ended up losing — negative, do it less.

But "how did things actually go" can be measured two ways, and both are flawed:

• Actual outcome only. Follow the real trajectory to the end and use what really happened. Honest — but drenched in noise: damage rolls, critical hits, the opponent's own luck. A perfect move followed by three bad rolls reads as a mistake.
• Critic only. Take one real step, then trust the value head's estimate of everything after. Smooth and low-noise — but only as good as the critic, and the critic is a small network trained on its own imperfect play. Its errors become your errors, systematically.

GAE — generalized advantage estimation — refuses to pick a side. It blends the two along the trajectory: trust short-horizon reality (the next few actual steps), plus the critic's estimate of the rest, with a knob λ controlling the blend. λ = 0 is pure critic; λ = 1 is pure actual outcome; this project runs λ = 0.95 (assign_returns() in src/selfplay/trainer.py) — mostly reality, smoothed by the critic. The critic also bootstraps credit backward through the game, which is how the very first decision — the pick-4 — receives usable credit without the terminal reward having to propagate raw through every intervening turn.

The result in this project: GAE was the single largest strength gain of the entire credit-assignment thread. In a controlled head-to-head, the GAE-trained policy rated Elo 80 vs 44 for the ablated (no-GAE, unshaped) run — +36 Elo (docs/roadmap.md, docs/architecture.md). And notice what the lever actually was: GAE added no new information. Same games, same rewards, same features. It reduced the variance of each update — less luck contaminating each gradient step, so every batch of games taught more. In data terms: you didn't collect new data, you fixed the aggregation so the same data stopped lying to you per-batch.

The instructive failure: the ally-KO penalty

One more story, because failed levers teach as much as successful ones. Problem: the policy would sometimes Earthquake — a spread move that hits everyone adjacent, partner included — while its own healthy ally stood next to the epicenter. Goal: stop KOing your own partner. The obvious RL-shaped fix: a −0.3 reward penalty on any transition where a chosen move KOs a healthy, exposed ally (ALLY_KO_PEN, wired through assign_returns).

It was tested properly: a controlled ablation — two from-scratch training runs, identical in every respect (same seed, same recipe) except that one had the penalty and one didn't. Both runs converged to roughly 18% ally-KO propensity. The penalty did nothing. The real improvement — down from the earlier default's 25% — had already come from something else entirely: fixing a bug in the allyHit feature, which had been telling the policy that single-target moves like Knock Off could somehow hit the partner (details in docs/regression.md). Perception fixed it; punishment didn't. The penalty code was reverted (docs/roadmap.md §7).

Why was it inert? Ally-KOs are rare. A −0.3 penalty on an event that appears in a small fraction of games produces a gradient that is negligible against the pressure of millions of ordinary decisions — the signal is simply too small a fraction of the total loss to steer anything. Two lessons worth keeping:

• Rare events need a big lever or a different mechanism. A much larger coefficient (risking distortion of legitimate Earthquake use), upweighting the rare transitions, or stepping outside RL entirely — masking the action or a decision-time rule, the way the fully-immune-attack blunder was fixed. Until one works, ally-KO stays report-only in the regression gate.
• You only know the lever did nothing because the ablation was controlled. Change one thing, keep the seed, compare. Without the paired run, the drop from 25% to 18% would have been credited to the penalty, the belief would have shipped, and the real cause — the feature fix — would have gone unrecorded. It's the same discipline as an A/B test with a held-out control: no control, no attribution.