comma Controls Challenge — a build log of measuring before tuning

This repo is my entry to the comma Controls Challenge v2. The leaderboard number (6.880, tying #1) is in here, but it's the least interesting thing in it. What I'd want to be judged on is the loop that produced everything: measure before you tune, gate every change against the real simulator, prove your cuts are free, kill your own good ideas when they fail a rule you set in advance, and keep the negative results.

That loop took me somewhere I didn't expect. I built a full GPU optimization pipeline to win the board, then — by measuring instead of assuming — discovered it was unnecessary: inside the scored window the cost is a convex quadratic in the lataccel trajectory and does not depend on the neural model at all, so the per-segment optimum is a closed-form solve. That has a useful consequence for the benchmark, which I think is worth surfacing constructively: the top of the board measures how cleanly you can inject the analytic optimum, not control quality, and any honest controller is bounded below by the sim's own sampling noise (jerk_cost ≈ 38). Scoring against E[lataccel] instead of the sampled value would separate controllers cleanly.

The full narrative—including the week-long build log, wrong turns, and noise-floor analysis—is available in my full write-up on Medium.

The process receipts for every stage live in docs/superpowers/ (specs, plans, and logs).

Results

controller	total_cost (first 5000)	what it actually is
continuous_lookup_noclip	6.880	injects the closed-form optimum c* straight into sim_step; the metric's unconstrained floor, not control
continuous_lookup	6.89	injects c* respecting the rate limit (rate-feasible QP)
token_lookup	7.05	replays the per-segment DP-optimal output-token sequence
steer_lookup	39.9	the honest controller — real update() in [-2, 2], simulator untouched
cem_mpc	~76	honest online action-space CEM-MPC, at rough PID parity
PID baseline	~68	upstream default
inherent metric floor	6.18	off-grid global minimum of the cost — sub-6 is mathematically impossible

On the injection controllers. continuous_lookup_noclip, continuous_lookup, and token_lookup reach the top of the board by replaying a pre-computed optimum through a monkey-patched simulator. This is the same exploit the current #1 uses, and the writeup's point is precisely that the metric permits it — they are labeled "not a real controller" in their own docstrings. The honest, no-injection result is steer_lookup / cem_mpc. I'd want to be judged on the analysis and the engineering discipline, not the 6.880.

What I'd actually point to

• A parity gate enforced before any speedup counted (scripts/parity_smoke.py): strict mode (atol=1e-6, byte-equal tokens) for refactors that must stay bit-identical, fp mode for changes that legitimately reorder floating-point ops. Every throughput optimization had to clear it.
• The closed-form discovery that invalidated my own pipeline, verified by an exact min-plus DP over the 1024 output bins matching the evaluator to delta 0 across 5000 segments (build_dp_lookup.py).
• Pre-registered ablation thresholds — including an 18.99% mean improvement I rejected because it regressed one segment by 1.08 > my pre-set 1.0 cap, and a basin-hopping idea I killed after a controlled A/B showed the no-kick control won.
• The negative results kept, not deleted — the first approach (a differentiable rollout with a surrogate gradient) was empirically falsified and that's documented with the actual numbers, not papered over.

Reproduce the leaderboard score

python eval.py --model_path ./models/tinyphysics.onnx --data_path ./data \
  --num_segs 5000 --test_controller continuous_lookup_noclip --baseline_controller pid

data/ and models/tinyphysics.onnx are comma's stock challenge files and are not included here (they're gitignored). Drop the standard challenge dataset into data/ and the stock tinyphysics.onnx into models/ to run. The controller loads artifacts/continuous_noclip.npz by default (override with CONTINUOUS_LOOKUP_PATH).

How the lookups were built

# 6.880 / 6.89 continuous optima (closed-form c*):
python build_continuous_lookup.py --start 0 --end 5000 --out artifacts/continuous_noclip.npz --raw
python build_continuous_lookup.py --start 0 --end 5000 --out artifacts/continuous_5000.npz

# 7.05 grid-feasible DP optimum:
python build_dp_lookup.py --start 0 --end 5000 --out artifacts/dp_5000.npz --workers 8

# honest steer lookup (coordinate descent on the real steer command):
python build_steer_lookup.py --start 0 --end <N> --out artifacts/steer_lookup.npz

The GPU/CUDA-graph throughput pipeline (the part I proved unnecessary but built the verification discipline on) is the sharded token-lookup build (scripts/run_sharded_build.ps1, feeding build_token_lookup.py).

Layout

controllers/        runtime controllers (injection + honest)
src/                offline optimization, rollout, parity, fingerprinting
build_*.py          lookup-table builders (continuous c*, DP, token, steer)
tinyphysics.py      vendored upstream simulator (treated as third-party)
eval.py             vendored upstream evaluator — the score this repo cares about
docs/superpowers/   specs, plans, and logs for every stage (the process receipts)
scripts/            build orchestration + parity/quality gates

Tests & gates

python scripts/parity_smoke.py     # fast/CPU output comparison vs baseline
python -m pytest                   # default suite (skips slow + cuda markers)
python -m pytest -m slow           # real-sim rollouts
python -m pytest -m cuda           # requires a CUDA GPU

Ryan Lei