Unit tests for turn-level credit assignment in agentic reinforcement learning.
AgentCreditBench scores credit estimators on tiny finite-horizon MDPs with exact policy-advantage oracles. It separates two questions: does an estimator identify which actions helped, and does it induce the intended policy gradient? The core runs on CPU with Python ≥3.10 and zero runtime dependencies.
Website · Reproduce the evidence · Detailed results and audits · Framework validation
On successful BAD → RECOVER trajectories, the recovery environment's exact
advantage gives the mistake −0.25 and the repair +0.50. Broadcasting the successful
return assigns +1 to both. The oracle distinguishes the mistake from the repair;
these broadcast credit values do not.
This selected-path result does not establish a biased expected gradient. Unsuccessful trajectories can offset the positive credit on repaired mistakes. The suite therefore reports both credit-identification and gradient metrics, including aggregates over all BAD actions. See the full recovery diagnostic.
The package is not yet published on PyPI. Install from a source checkout:
git clone https://github.com/hectopascal/agent-credit-bench.git
cd agent-credit-bench
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"from agent_credit_bench import UniformPolicy, run_benchmark
from agent_credit_bench.envs import DelayedEffectEnv
from agent_credit_bench.estimators import BatchCenteredBroadcast
result = run_benchmark(
mdp=DelayedEffectEnv(horizon=16),
policy=UniformPolicy(),
estimator=BatchCenteredBroadcast(),
batch_size=1_000,
seeds=range(10),
)
print(result.mean_gradient_cosine, result.seed_metrics[0].leakage_ratio)
print(result.seed_metrics[0].normalized_squared_gradient_error)A custom estimator implements estimate(context) and returns one credit per
sampled action. Estimator interface and randomness contract.
| Evidence | What actually runs | What it establishes |
|---|---|---|
| Exact oracle and differential tests | Backward induction, independent rational checks, enumerated return derivatives | Tabular values and gradients under the supplied policy |
| Frozen fault study | 6 faulty variants and 4 valid controls across 42 cases | All 6 faults detected; no false alarms on these 4 controls; the full-reference comparator also detects all 6 |
| Installed verl / OpenRLHF | Advantage functions / advantage pipeline; documented CPU import stubs for unused OpenRLHF GPU paths | Credit-computation conformance at pinned versions |
| TRL | Source-fingerprinted transcription plus installed standard-deviation helper | Agreement with the pinned trainer math |
| GiGPO / rLLM / prime-rl | Complete source module or unchanged extracted definitions with controlled inputs | Source-level estimator and routing behavior, not complete training runs |
| rLLM runtime regression | Pinned helper with real PyTorch; separately tested proposed patch | Reproduces an empty-batch crash and verifies the candidate helper fix |
The synthetic fault corpus is author-designed, not independently curated. A passing runtime-regression test means the known pinned defect reproduced; it does not certify the upstream implementation as bug-free. Optional tests that skip are not compatibility evidence.
python scripts/reproduce.py --out /tmp/acb-evidenceNo installation is needed for this command. It regenerates the frozen fault
study, core estimator matrix, and full variable-horizon frontier; compares
saved tables; and writes logs plus a source/output hash manifest. The output
directory must be new. Use --suite core for the shorter run.
The evidence workflow also builds a source archive and reproduces results from its extracted contents. Separate source conformance checks fetch fingerprinted upstream files and reject skipped tests. Environment setup, release packaging, tolerances, and commands.
TurnLOO can improve per-turn calibration while increasing finite-batch gradient MSE in some configurations. The default 90-configuration frontier now uses exhaustive expectations for 45 cells and paired, pointwise 95% bootstrap intervals for the other 45; seven of those sampled comparisons are unresolved. These intervals do not provide simultaneous coverage for the whole frontier.
All 54,000 seed errors · Paired comparisons and exact expectations · Statistical protocol
- DelayedEffectEnv: a consequential first action followed by irrelevant turns.
- RecoveryEnv: an early mistake that a later action can repair.
- VariableHorizonEnv: STOP/CONTINUE decisions control which later turns exist.
Included estimators: exact advantage, outcome broadcast, leave-one-out trajectory centering, TurnLOO, GRPO-style normalization, GiGPO-style grouping, Monte Carlo advantage, and adapters for external return-to-credit functions. Definitions and results.
The verl bridge and verifiers bridge can collect model trajectories. Their oracle is exact for a fitted tabular Markov projection, not the original history-conditioned language-model policy.
Passing establishes agreement with the tested tabular contracts under the recorded policies and configurations. It does not establish downstream learning performance. Core results use undiscounted return (γ=1) and tabular softmax policy gradients. Function approximation, token-level objectives, PPO clipping, KL penalties, off-policy learning, partial observability, and learned critics remain outside the core diagnostics. Numerical and integration boundaries.
The archived software citation below refers to v0.5.0; development changes in this checkout are not a newly published DOI release.
@software{yiyan2026agentcreditbench,
author = {Yi Yan Ng},
title = {{AgentCreditBench}: A Conformance-Test Suite for Turn-Level Credit Estimators},
year = {2026},
version = {0.5.0},
doi = {10.5281/zenodo.22178939},
url = {https://github.com/hectopascal/agent-credit-bench},
license = {MIT}
}