Research · Recursive Self-Improvement
How to Improve an AI Without Letting It Fool Itself
Recursive self-improvement, Goodhart's law, and why your evaluator is really a control system — explained through a mechanical watch.
Here is a game that sounds too good to be true. Take an AI agent. Give it a way to grade its own work. Then let it rewrite itself to chase a higher grade, over and over. Each round it gets a little better, and the better version is a little better at improving itself. That compounding loop is recursive self-improvement (RSI)1, and in miniature it already works. The everyday machinery is called Automatic Prompt Optimization (APO)2: propose variations of an agent's instructions, score each one with a judge, keep the winners, repeat.
There's a catch, and it's the whole story. An optimizer pointed at a score does not learn to be good. It learns to score well.3 Those are the same thing right up until they aren't.
The trap has a name
It's Goodhart's law4: when a measure becomes a target, it stops being a good measure. Point a relentless optimizer at a fixed judge and it will find the judge's blind spots faster than the judge can cover them. The reported score sails upward while the real quality it was standing in for quietly flatlines.
This isn't hypothetical, and it isn't exotic. It's the default outcome of any self-improvement loop with a frozen evaluator. So the real question isn't “how do we make the agent better?” We can already do that. The question is:
How does a system improve itself using a judge that the very same optimization is trying to game?
Stop treating the evaluator as a ruler
The usual mental model is that the evaluator is a ruler — a fixed, passive instrument you hold up against the work. That framing is exactly what gets you Goodharted. The fix is to borrow a lens from control theory5. There are two feedback loops here, and they do opposite jobs:
- The optimizer is a reinforcing loop— positive feedback. It's the engine that drives capability up, and left alone it runs away.
- The evaluator is a balancing loop — negative feedback. Its job is to give the reinforcing loop a goal and to anchor that goal to ground truth.
Stable recursive self-improvement is not a model-quality problem. It's a control problem: keep the balancing loop robust, well-anchored, and — this is the part everyone misses — faster at closing holes than the reinforcing loop is at finding them.
The whole thing is a mechanical watch
The cleanest picture I've found is a mechanical watch. A watch turns a raw, run-away energy source into something that keeps honest time, using nothing but nested loops and one contact with reality.
The fast gearis your inner loop: APO spinning every generation, tuning the agent's skill θ. The slow gears are the outer loop, where you improve the evaluator φ itself — rarely, but with the most leverage. The escapement is the audit gate that lets progress out one controlled tick at a time. And the pendulum is the human anchor: a locked set of human-labeled examples that the fast loop can never see or touch. A watch only keeps time because the pendulum is calibrated against real, external time. Take the pendulum away and you have a beautiful machine that is confidently, precisely wrong.
Three ways it breaks
Once you see it as coupled feedback loops, the failure modes aren't mysterious. There are exactly three, and each maps to a specific pathology and a specific fix.
| Failure | What's happening | The fix |
|---|---|---|
| Reward hacking6 | The reinforcing loop drills a crack in the ruler. | Weld the ruler shut: anonymized, held-out, locked evaluation. |
| Oscillation / collapse | The balancing loop's delay outruns the optimizer's speed. | Refresh the evaluator faster than the optimizer finds exploits. |
| Plateau | The reinforcing loop hits a fixed ceiling — the ruler's own quality. | Co-evolve the evaluator so the ceiling rises too. |
The one rule to remember
Here is the rule I'd tattoo on the loop if I could: slow evolution is not the same as slow feedback.The evaluator's weights may only update every few generations — that's fine. But the audit gate, the escapement, must fire every single generation.If the fast loop runs many rounds un-audited, it eats the ruler's blind spots before the slow refresh ever arrives, and the system overshoots, oscillates, or collapses.
You can even see it in the stability condition. Model the error dynamics as
ė(t) = g·e(t) − k·e(t−τ)
where g is how fast the optimizer amplifies error, k is the corrective strength of the evaluator, and τ is the audit delay in generations. The loop stays stable only while τ < τc. Push the un-audited delay past that critical value and you get overshoot → oscillation → divergence. Auditing every generation is just the engineering move that keeps τ small.
One line for the team: optimization can be slow. Monitoring cannot.
Does any of this actually work?
Yes — and the way it broke first is the best evidence it's real. I built an APO system and pointed it at banking777, a 77-way intent classifier, but anonymized: every intent label replaced by an opaque token so the model couldn't lean on names it saw in pretraining. The judge scored on a locked test set read exactly once, at the very end.
Locked test accuracy went from 0.06 to 0.76. The optimizer had induced clean descriptions for 21 of 25 intents purely from labeled failures, never once seeing a label name in its prompt. Because the test set was welded shut, that +70 points is real signal, not the optimizer cheating its way up.
How do I know? Because the firstrun leaked. An example ID accidentally exposed the labels, and the optimizer found it instantly — the score shot up on a crack in the ruler rather than on genuine skill. That's the reinforcing loop drilling the balancing loop, on video, exactly as the theory predicts. Anonymize and lock the evaluator and the same optimizer produces a real gain. Same engine; the difference is entirely in whether the ruler can be gamed.
The honest limit
I'll say the uncomfortable part out loud, because the framing demands it. With no continuous, independent contact with reality, a self-improving system can only ever optimize a proxyof its own quality. “True convergence” means convergence relative to the best oracle you've got. Unbounded self-improvement — the system getting endlessly better with no external anchor — is unfalsifiable at the very top of the hierarchy. It's a conjecture, not a theorem.
Which is why the highest-leverage move in the whole design isn't a bigger model. It's a better ruler. If you want the ceiling to keep rising, the evaluator has to improve alongside the agent — anchored, every step, to something real.
That's the thesis I keep coming back to: RSI is a control problem wearing a machine learning costume. Get the loops right — a fast engine, a slow honest ruler, an escapement that never sleeps, and a pendulum tied to the real world — and self-improvement is safe. Get them wrong and you've built a watch with no pendulum: fast, precise, and telling you exactly the wrong time.
References
- J. Schmidhuber. “Gödel Machines: Fully Self-Referential Optimal Universal Self-Improvers.” In Artificial General Intelligence, Springer, 2007.
- R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, M. Zeng. “Automatic Prompt Optimization with ‘Gradient Descent’ and Beam Search.” EMNLP, 2023. arXiv:2305.03495.
- D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, D. Mané. “Concrete Problems in AI Safety.” 2016. arXiv:1606.06565.
- M. Strathern. “‘Improving ratings’: audit in the British University system.” European Review5(3): 305–321, 1997 — the now-standard phrasing of C. Goodhart's law.
- D. H. Meadows. Thinking in Systems: A Primer. Chelsea Green, 2008 — reinforcing and balancing feedback loops.
- V. Krakovna et al. “Specification gaming: the flip side of AI ingenuity.” DeepMind, 2020.
- I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, I. Vulić. “Efficient Intent Detection with Dual Sentence Encoders.” NLP4ConvAI workshop, ACL 2020 — introduces the banking77 dataset.