An RL environment that tests whether language models can diagnose bugs that do not announce themselves.
Most code-repair benchmarks use bugs that raise a traceback or fail a test. Bugs in reinforcement learning training code often do neither. The program runs, the loss goes down, nothing crashes, and the policy just quietly fails to learn. That failure mode is what this repository measures.
Here is a real instance from the bug bank. One line changes in a standard PPO implementation:
- logratio = newlogprob - b_logprobs[mb_inds]
+ _, old_logprob, _, _ = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds])
+ logratio = newlogprob - old_logprobThe "old" log-probability is now recomputed from the current policy instead of being read from the rollout buffer. Both terms are the same function evaluated at the same weights, so they cancel exactly. The PPO ratio is identically 1.0 and the policy gradient is exactly zero, verified with a standalone autograd check.
What the developer sees:
| Signal | Behavior |
|---|---|
| Exceptions | None |
| Test suite | Passes |
| Value loss | Decreases normally |
| Entropy | Evolves normally |
| Mean episodic return | 22.3 instead of 175.5 |
Nothing points at the broken line. The only evidence is a number that is lower than it should be, and you only know it should be higher if you already know what good looks like. An agent has to reason from training dynamics back to a mechanism. That is the skill this benchmark scores.
(Numbers: dead_surrogate_v1__seed0, CartPole-v1, 40k timesteps, 3 seeds, from
calibration/baselines.json.)
| What it is | A containerized eval harness. The agent gets a workspace, a run_training tool, metrics tools, and a turn budget |
| Scoring | Fully programmatic. No LLM in the reward path, ever |
| Substrate | PPO on CartPole-v1. Two bases: vendored CleanRL, plus a de-memorized 7-module reimplementation |
| Committed evidence | 63 full episodes, transcripts and per-episode results in eval/ |
| Models evaluated | claude-sonnet-4-5, claude-haiku-4-5 |
| Headline result | With code access, models fix the bug ~95% of the time and localize it 100% of the time. With only training curves, component identification drops to 67% and 33% |
| Main negative result | The v0 task substrate was too easy, and the reason is interesting enough to be the project's primary output so far |
| Where it is going | From eval to training environment, then toward a self-directed RL-tuning loop. See Direction |
Every result below traces to a committed file in eval/results/. 63 episodes
total, every one status: OK with hack_attempt: false.
dead_surrogate_v1, 5 instances x 3 episode seeds per model. Outcome is
normalized recovered performance. Localization is whether the model identified
the correct line.
| Model | n | Outcome | Localization |
|---|---|---|---|
claude-haiku-4-5 |
15 | 0.950 +/- 0.008 | 1.000 +/- 0.000 |
claude-sonnet-4-5 |
15 | 0.950 +/- 0.008 | 1.000 +/- 0.000 |
Localization has zero variance in both groups. Every single episode found and
fixed exactly the right line. claude-opus-4-8 was skipped after sonnet came
back statistically indistinguishable from haiku.
The agent sees only training metrics and a symptom string, and submits a diagnosis naming the responsible component. Same instances, same seeds.
| Model | n | Component match | Queried metrics before submitting |
|---|---|---|---|
claude-haiku-4-5 |
15 | 0.667 +/- 0.122 | 1.000 |
claude-sonnet-4-5 |
15 | 0.333 +/- 0.122 | 1.000 |
The comparison that matters is arm A localization vs arm D component match. Both ask "did the model find the right thing," not "did it fix it." Localization is a flat 1.000 with code access. Component match drops to 0.667 and 0.333 when the code is taken away. Removing file access produces a real, measurable gap on a bug that is otherwise solved at ceiling.
Statistics discipline. At n=15, the standard error on a proportion near 50%
is sqrt(0.5 x 0.5 / 15) = ~0.129, about 13 points. Differences under roughly 26
points are not reliable. The arm A vs arm D gaps (33 and 67 points) clear that
bar. The haiku vs sonnet difference within arm D (33 points) sits right at the
edge of it, and I do not read it as a capability ranking.
A nuance worth recording. Both models' wrong answers cluster on
advantage_estimation rather than scattering randomly, and that is not a careless
guess. A frozen policy alongside a normally-learning value function (zero KL, zero
clip fraction, near-zero policy loss, rising explained variance) is genuinely
consistent with either a dead policy-gradient term (the actual bug) or collapsed
advantages feeding that same term. The single ground-truth label may be
underdetermined by the metrics this bug exposes. That is a benchmark design
problem to fix, not evidence that sonnet reasoned badly.
The first version of the benchmark was too easy, and it took three attempts to work out why.
| Attempt | Change | Result |
|---|---|---|
| 1 | logratio = newlogprob - newlogprob, a literal self-subtraction |
Solved 3/3, sometimes without running training at all |
| 2 | Same mechanism, visual tell removed (redundant fresh forward pass). Calibration came out byte-identical | Solved 3/3 |
| 3 | Rebuilt the base as a 7-module package with de-CleanRL'd naming and structure, verified equivalent to the legacy base within seed noise (176.8 vs 175.5) | Legacy 3/3, modular 3/3, delta 0.000 |
Diagnosis. De-memorizing the code did not move the needle. The model is not
diffing the workspace against a memorized reference file. It recognizes a
well-understood PPO mistake from domain knowledge of the algorithm, independent of
code layout or naming. It needed far more exploration on the modular base (8
read_file calls across modules vs 1-2 on the single legacy file) and still found
the bug every time. Some bug classes are recognizable on sight to anything that
understands the algorithm, and cosmetic restructuring does not change that.
Four omission bugs were built on the modular base and calibrated against the same 40k-timestep CartPole-v1 budget, 3 seeds each. All four improved training instead of degrading it, and all four were rejected.
| Bug | Mechanism removed | Clean | Broken | Margin | Threshold | Result |
|---|---|---|---|---|---|---|
grad_clip_omitted_v1 |
Gradient clipping | 176.8 | 318.0 | -141.2 | 140.0 | Rejected |
entropy_omitted_v1 |
Entropy bonus in the total loss | 176.8 | 208.8 | -32.0 | 37.8 | Rejected |
adv_norm_omitted_v1 |
Advantage normalization | 176.8 | 214.0 | -37.1 | 51.6 | Rejected |
stale_bootstrap_v1 |
Fresh bootstrap value | 176.8 | 211.4 | -34.6 | 85.8 | Rejected |
Margin is clean minus broken. A bug must clear a positive margin exceeding 3x the larger seed standard deviation to be accepted. All four margins are negative: broken beat clean at every seed tested.
This is the expected result, not a fluke. Gradient clipping, the entropy bonus, and advantage normalization are variance-reduction machinery. They pay off on hard, unstable tasks and cost throughput on easy, well-scaled ones. CartPole-v1 has two actions, dense uniform reward, no real exploration problem, and advantages already at reasonable scale. Removing machinery that was never doing any work cannot degrade the task. It just lets the optimizer move faster. Longer budgets do not rescue this: CartPole-v1 caps return at 500, so both arms saturate and the ceiling hides degradation rather than revealing it.
This produced a design rule that now governs every new bug (invariant 6 below): ablate the mechanism on the clean base first and confirm it measurably hurts, before writing the bug at all.
The substrate is too easy, not the bugs wrong. A well-known algorithm mistake is recognizable regardless of memorization or code layout, and an omission cannot break a task where the omitted piece was never load-bearing. The fix is harder environments where the affected mechanisms genuinely carry weight, which is exactly where compute becomes the binding constraint.
Everything above is an eval. It measures. The direction I am moving in is an environment that trains, and then a loop that closes on itself.
Step 1, planned: a training environment. The scoring here was designed from day one to support RL fine-tuning against the bench. That is why there is no LLM in the reward path, why the outcome signal is dense rather than binary, and why instances are procedurally generated so a train/test split by bug mechanism (not by instance) is possible. The claim "models are bad at diagnosing silent RL bugs" is interesting. The claim "here is a verifiable-reward environment that measurably improves them on held-out bug types" is a different tier of result.
Step 2, the actual goal: close the loop. What I want is a model that does not just fix a broken training run but runs the whole empirical loop itself on a hard RL problem. Propose a training configuration, launch it, read the curves, form a hypothesis about what the dynamics imply, change one thing, iterate to a working policy, all without a human interpreting the plots. Diagnosing a silent bug is the smallest honest unit of that skill, which is why I started here. Arm D (diagnosis from training dynamics alone, no code access) is the piece of the current harness that most directly rehearses it.
The honest constraint: compute. This is why the results above are on
CartPole. Both findings say the same thing, that CartPole is too easy to be an
interesting substrate, and both point at environments where stability machinery is
genuinely load-bearing. Those environments are exactly the ones that cost real
wall-clock to train. A quadcopter base (drone_v1, adapted from my own
MIT-licensed environment, so no memorized reference exists) is specified in
tasks/roadmap.md, along with the mitigations I would apply:
set the budget from divergence rather than convergence, batch the physics,
shrink the task rather than the fidelity, and cache training runs by workspace
hash. It is not built. Nothing in this repository trains on anything but
CartPole-v1 today, and I would rather say that than imply otherwise.
Compact summary. Full detail in tasks/roadmap.md and
tasks/hardness-v1.md.
Bases. legacy_cleanrl (vendored, unmodified CleanRL PPO) is the easy tier
and memorization control. modular_v1 (7-module reimplementation, de-CleanRL'd
naming) is built. drone_v1 is planned, not built.
Bug classes. wrong_line (one incorrect line), omission (a missing
operation, nothing on screen to inspect), interaction (every line correct in
isolation, wrong only in combination with a config value or another module),
statistical (visible only in the numbers across iterations).
Observation arms.
| Arm | Access | Implemented |
|---|---|---|
| A | Files, run_training, stdout |
Yes |
| B | A plus get_metrics, list_metric_keys |
Yes |
| C | B plus rendered plot images | No |
| D | run_training and metrics only, no file access |
Yes |
Difficulty gradient. A benchmark where everything scores 0 is as uninformative as one where everything scores 1.
| Tier | Target solve rate | Typical composition |
|---|---|---|
| Easy | 80-100% | legacy_cleanrl, wrong_line |
| Medium | 40-70% | Unmemorized base, wrong_line or omission |
| Hard | 10-40% | Unmemorized base, interaction or statistical |
No change may violate these.
- No LLM anywhere in the reward path. Every scoring component is computable by a deterministic script.
- Determinism. Same instance plus same seed gives the same score. Python,
NumPy and Torch seeded;
CUBLAS_WORKSPACE_CONFIGpinned. - The agent cannot touch the scorer. Scoring lives outside the writable
workspace, hash-checked before and after every episode. A mismatch marks the
episode
INVALID. - No network access inside the agent container.
- A bug that does not degrade performance is not a task. The clean-vs-broken gap must exceed 3x the larger seed standard deviation, or it is rejected.
- A bug class is only valid on a base where the affected mechanism is load-bearing. Ablate first, confirm it degrades the clean baseline, then write the patch. (This rule exists because of Finding 2.)
- Difficulty comes from the bug and the search space, never from unreadable code. Bases stay idiomatic. Obfuscation is not a valid difficulty source.
- Log every trajectory in full. Transcripts are a primary output, not a byproduct.
| Phase | What | Status |
|---|---|---|
| v0 | Harness: container, tools, episode loop, scoring | Done |
| v0 | Calibration (3 bug types) | Done, 11/11 instances accepted |
| v0 | Smoke eval difficulty checkpoint | Done, target missed twice, see Finding 1 |
| v1 | Modular de-memorized base, legacy-vs-modular checkpoint | Done, see Finding 1 |
| v1 | Omission-bug ablations on the modular base | Done, 0/4 accepted, see Finding 2 |
| sprint | Multi-model arm A sweep | Done |
| sprint | Arm D (diagnosis-only) harness and sweep | Done |
| sprint | Qualitative read of arm D transcripts | Not started |
| sprint | eval/analyze.py |
Not started |
| next | Unmemorized, load-bearing base (drone_v1) |
Not started, compute-gated |
| next | Training environment (RL fine-tuning against the bench) | Not started |
Nothing is currently running. "Not started" rows are exactly that, not work stalled mid-flight.
Every directory below exists in the pushed tree.
base/legacy_cleanrl/ vendored, unmodified CleanRL PPO, the memorization control
base/modular_v1/ 7-module reimplementation, same algorithm, de-CleanRL'd naming
bugs/ registry.yaml + patches/
calibration/ build_baselines.py + committed baselines.json
harness/ container lifecycle, tools, episode loop, metrics store, model adapters
scoring/ outcome, localization, integrity/hack-detection, score_episode entrypoint
tests/ pytest suite (fast tests + a `slow` marker for real-training tests)
eval/ transcripts/ and results/, 63 committed episodes
tasks/ spec and planning documents
make install # pip install -e ".[dev]"
make test-fast # everything except real-training tests
make test # full suite, real Docker training runs
python calibration/build_baselines.py # regenerate calibration/baselines.json
make repro is not implemented. It exits with an error pointing back to this
file rather than pretending to run a sweep that does not exist end to end.
Running a live episode needs ANTHROPIC_API_KEY (via
harness.models.AnthropicAdapter). Keys live in a gitignored .env and are never
committed. There is no pre-commit hook scanning for leaked keys yet. Every commit
has been checked by hand, but an automated grep -r "sk-ant-" hook would be a
reasonable addition.
| Document | Covers |
|---|---|
tasks/tasks-list.md |
v0 spec: harness and scoring design, still authoritative for those pieces |
tasks/hardness-v1.md |
v1 difficulty redesign, bug classes and levers |
tasks/roadmap.md |
Full project plan, phases 0 through 5, including the training environment |
tasks/weekend-sprint.md |
The scoped sprint that produced the arm A vs arm D results |
MIT. The vendored CleanRL base (base/legacy_cleanrl/) is MIT and unmodified.